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Part  One 

Test  and  Evaluation  in  System  Acquisition 

Chapter  1 
Introduction 

1-1.  General 

Developing  and  fielding  Army  systems  that  achieve  the  required 
performance  and  are  operationally  effective  and  suitable 
represent  significant  challenges  to  all  involved  in  the  system 
acquisition  process.  The  procedures  and  guidelines  in  this 
pamphlet- 

a.  Apply  to  all  systems  developed  and  managed  under  the 
auspices  of  AR  70-1;  these  systems  are  referred  to  as  materiel 
systems  in  this  pamphlet.  This  category  includes  systems  that 
contain  computer  hardware  and  software  specifically  designed, 
configured,  and  acquired  as  an  integral  element  of  the  system  and 
needed  so  that  the  system  can  fully  perform  its  mission  (Materiel 
System  Computer  Resources) . 

b.  Apply  to  all  systems  developed  and  managed  under  the 
auspices  of  AR  25-3;  these  systems  are  referred  to  as  information 
systems  in  this  pamphlet.  As  used  in  this  pamphlet,  the  term 
information  system  applies  to  systems  that  evolve,  are  acquired, 
or  are  developed  and  that  incorporate  information  technology.  It 
applies  to  all  information  systems  of  the  five  information 
mission  area  (IMA)  disciplines  not  developed  and  managed  under  AR 
70-1.  It  encompasses  automated  information  systems,  except  those 
that  are  used  exclusively  for  cryptological  activities  or  those 
acquired  under  the  National  Foreign  Intelligence  Program  for 
operational  support  of  intelligence  and  electronic  warfare 
systems . 

1-2 .  Scope 

One  of  the  fundamental  elements  of  the  acquisition  process  is 
test  and  evaluation  (T&E) .  The  primary  objective  of  T&E  in 
support  of  the  acquisition  process  is  the  verification  of 
developmental  and  operational  goals  and  objectives.  The 
structuring  and  execution  of  an  effective  T&E  program  is 
absolutely  essential  to  the  acquisition  and  fielding  of  Army 
systems  which  meet  the  user's  requirements.  There  are  many 
elements  integral  to  a  successful  T&E  program.  This  pamphlet 
provides  procedural  guidance  to  implement  the  policies  in  AR 
73-1,  Test  and  Evaluation  Policy,  with  regard  to  planning, 
executing,  and  reporting  T&E  in  support  of  the  acquisition 
process.  Specifically,  this  pamphlet  provides  procedural 
guidance  in  the  following  areas; 
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a.  Establishing  and  conducting  a  Test  Integration  Working 
Group  (TIWG) . 

b.  Preparing  and  staffing  Critical  Operational  Issues  and 
Criteria  (COIC) . 

c.  Preparing  and  staffing  a  Test  and  Evaluation  Master  Plan 
(TEMP) . 

d.  Planning,  executing,  and  reporting  Developmental  Test  and 
Evaluation. 

e.  Planning,  executing,  and  reporting  Operational  Test  and 
Evaluation. 

f.  Structuring  a  Live  Fire  T&E  program. 

g.  Addressing  reliability,  availability,  maintainability, 
integrated  logistic  support,  manprint,  threat,  survivability, 
compatibility,  interoperability,  and  modeling  and  simulation  in 
support  of  T&E. 

h.  Planning  and  conducting  T&E  of  software  as  an  integral 
part  of  the  overall  T&E  program  of  a  system. 

i.  Acquisition  of  Instrumentation,  Targets  and  Threat 
Simulators  (ITTS)  in  support  of  T&E. 

1-3.  References 

Required  and  related  publications  are  listed  in  Appendix  A. 


1-4.  Explanation  of  Abbreviations  and  Terms 
Abbreviations  and  special  terms  used  in  this  pamphlet  are 
explained  in  the  glossary. 
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Chapter  2 

Army  Test  and  Evaluation  Philosophy 
2-1.  General 

a.  Test  and  Evaluation  is  an  essential  part  of  the 
development  and  fielding  of  all  Army  systems.  The  information^ 
generated  as  a  result  of  T&E  influences  every  action  taken  during 
the  system  acquisition  process.  Defense  Acquisition  Boards 
(DABs) ,  Army  System  Acquisition  Review  Councils  (ASARCs) ,  Major 
Automated  Information  Systems  Review  Councils  (MAISRCs) ,  and 
In-Process  Reviews  (IPRs)  use  T&E  reports  generated  from  test 
data  and  analyses  to  assist  in  major  milestone  decisions. 
Developers  require  test  data  to  provide  feedback  on  design^ 
elements  in  order  to  assure  adequate  progress  towards  meeting 
user  requirements.  Contractors  use  T&E  information  to  ensure 
conformance  to  technical  data  packages,  and  to  detect^ 
manufacturing  or  quality  deficiencies.  Finally,  T&E  information 
can  provide  the  confidence  that  users  of  fielded  systems  must 
have  relative  to  their  systems  performing  as  required.  The 
importance  of  structuring  a  sound  test  and  evaluation  program 
during  the  system  acquisition  process  cannot  be  over-emphasized. 

b.  Army  T&E  policy  provides  the  flexibility  to  allow  each 
acquisition  program  to  tailor  a  T&E  strategy  to  achieve  maximum 
support  to  the  program  (See  Chapter  6).  T&E  strategies  must  be 
generated  concurrent  with  the  acquisition  strategy  to  assure  that 
T&E  is  an  integral  part  of  the  acquisition  program.  Efficient 
T&E  strategies  that  are  fully  integrated  into  acquisition 
programs  will  effectively  support  event— driven  acquisition 
philosophies. 

c.  Modeling  and  simulation  will  be  considered  to  support  the 
technical  and  operational  T&E  of  all  systems  (hardware  and 
software)  as  they  proceed  through  the  life  cycle.  Use  of  models 
and  simulations  will  include,  but  not  be  limited  to,  the 
identification  of  test  parameters  and  drivers  for  field  tests; 
determination  of  high  risk  areas;  prediction  of  test  results; 
assisting  in  the  allocation  of  scarce  test  resources;  and  the 
assessment  of  system  capabilities  in  situations  which  cannot  be 
tested  due  to  safety,  cost,  or  other  constraints.  The  extent  of 
the  use  of  modeling  and  simulation;  whether  existing  models  and 
simulations  will  be  used  or  new  ones  will  be  developed;  status  of 
models  and  simulations  verification,  validation,  and 
accreditation;  and  the  degree  to  which  models  and  simulations 
will  augment  test  data  to  assist  in  system  evaluations  and 
assessments  will  be  documented  in  the  Test  and  evaluation  Master 
Plan  (TEMP) .  Models  and  simulations  used  for  T&E  must  be 
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accredited  and  validated  prior  to  their  use  for  extrapolation  or 
predicting  system  performance  (including  software,  hardware,  man- 
in-loop)  (See  Chapter  16) . 

d.  Software  and  computer  resources  are  essential  components 
of  both  materiel  and  information  systems.  Software  T&E  for  both 
of  these  categories  of  systems  is  accomplished  within  the  context 
of  the  overall  system  development  and  test  program.  Criteria  for 
evaluating  progress  and  risk,  including  metrics,  will  be 
established  to  facilitate  determining  how  well  the  software 
supports  the  goals  of  system  effectiveness  and  suitability. 
Commonality  in  terms  and  T&E  approaches  between  materiel  and 
information  systems  will  be  emphasized  (See  Part  Seven) . 

2-2.  Basic  Test  and  Evaluation  Elements 

Army  T&E  consists  of  several  basic  elements  that  are  essential  in 
the  development  and  conduct  of  meaningful  T&E.  Each  of  these 
elements  will  be  discussed  in  detail  in  this  pamphlet.  These 
basic  elements  are: 

a.  Test  Integration  Working  Group  (TIWG) .  The  TIWG  is  the 
cornerstone  upon  which  a  smart,  effective,  T&E  strategy  is  built. 
The  TIWG  consists  of  members  of  the  T&E  community,  and  has  the 
responsibility  of  coordinating  and  integrating  all  test  and 
evaluation  planning  and  scheduling,  identifying  and  resolving  T&E 
problem  areas,  assuring  accurate  documentation  of  the  T&E 
strategy  in  the  Test  and  Evaluation  Master  Plan  (TEMP) ,  and 
assuring  that  all  Army  agencies  involved  in  the  T&E  program  are 
working  towards  a  common  goal.  The  TIWG  members  are  the  key 
players  in  the  T&E  program,  and  collectively  structure,  document, 
and  execute  the  T&E  program  (See  Chapter  8) 

b.  Test  and  Evaluation  Master  Plan  (TEMP) .  The  TEMP  is  the 
basic  planning  document  for  a  system's  life  cycle  T&E.  It  is 
required  for  all  acquisition  programs.  The  Program  Manager  (PM) 
or  Materiel  Developer  (MATDEV)  is  responsible  for  the  TEMP, 
however  all  TIWG  members  contribute  to  its  development  and 
maintenance.  The  TEMP  describes  in  broad  terms  what  testing  is 
required,  who  will  perform  the  testing,  what  resources  will  be 
needed  to  conduct  the  testing,  and  how  the  evaluation  will  be 
performed.  Upon  approval  by  the  appropriate  authority,  the  TEMP 
serves  as  a  contract  between  the  PM/MATDEV  and  the  T&E  community 
relative  to  the  execution  of  the  T&E  strategy  (See  Part  Two) . 

c.  Independent  Evaluations  and  Assessments.  Critical  to  the 
decision  making  process  is  the  availability  of  unbiased, 
objective  evaluations  and  assessments  of  a  system's  capabilities. 
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This  is  achieved  by  the  use  of  evaluators  and  assessors  which  are 
independent  from  the  development  community.  The  Army  T&E 
community  has  both  developmental  and  operational  independent^ 
evaluators  and  assessors.  AR  73-1  indicates  which  T&E  agencies 
have  independent  evaluation  or  assessment  responsibilities,  and 
Chapter  3  of  Part  One  further  explains  the  roles  and  missions  of 
the  independent  evaluators  and  assessors.  Parts  Four  and  Five 
discuss  in  greater  detail  independent  evaluation  and  assessment 
procedures  for  the  primary  Army  evaluators  and  assessors. 

d.  Developmental  and  Operational  Testing.  Much  of  the 
information  upon  which  independent  evaluations  are  based 
consists  of  data  generated  from  testing.  The  two  types  of  tests 
performed  within  the  Army  are  developmental  tests  and  operational 
tests.  Developmental  testing  is  performed  in  controlled 
environments  by  specially  trained  individuals  to  assess  the 
adequacy  of  the  system  design,  to  determine  compliance  with 
system  specifications  and  technical  parameters,  and  to  determine 
how  safe  the  system  is  for  operation  by  user  troops  and 
civilians.  Operational  testing  is  performed  in  realistic 
operational  environments  with  typical  user  personnel  to  assist  in 
the  determination  of  operational  effectiveness  and  suitability  of 
the  system.  Both  developmental  and  operational  testing  must 
address  all  system  components  (hardware,  software  and  human 
interfaces)  that  are  critical  to  the  achievement  and 
demonstration  of  contract  technical  performance  specifications 
and  minimum  acceptable  operational  performance  requirements 
specified  in  the  Operational  Requirements  Document  (ORD)  or 
Functional  Description  (FD) .  Combined  developmental  and 
operational  testing  should  be  considered  when  there  are  time  and 
cost  savings,  however  a  final  independent  phase  of  operational 
testing  is  required  for  beyond  low-rate  initial  production 
decisions.  Parts  Four,  Five,  Six,  and  Seven  discuss  in  greater 
detail  the  procedures  for  planning,  executing,  and  reporting 
developmental  and  operational  testing  and  evaluation. 

e.  Operational  issues  and  criteria.  There  are  two  types  of 
operational  issues  and  criteria  applicable  to  the  Operational 
Test  and  Evaluation  (OT&E)  process.  Critical  Operational  Issues 
and  Criteria  (COIC)  define  what  is  operationally  adequate  to 
proceed  into  full  production.  COIC  are  developed  by  the  combat 
developer  for  materiel  systems  and  theater/tactical  Information 
Mission  Area  (IMA)  systems  and  by  the  Functional  Proponent  (FP) 
for  strategic  and  sustaining  base  IMA  systems.  COIC  are  included 
in  Part  IV  of  the  TEMP.  Additional  Qperational  Issues  and 
Criteria  (AOIC)  provide  for  complete  and  comprehensive 
operational  evaluation  of  the  system.  AOIC  are  developed  by  the 
independent  operational  evaluator,  and  included  in  the  Test  and 
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Evaluation  Plan  (TEP)  along  with  the  COIC.  AOIC  complement  and 
supplement  the  COIC.  See  Part  Three  for  detailed  COIC  guidance. 
See  Part  Five  for  detailed  AOIC  guidance. 

2-3.  Continuous  Evaluation 

Continuous  Evaluation  (CE)  is  the  process  that  provides  a 
continuous  flow  of  T&E  information  on  the  capabilities  of  a 
system  to  all  levels  of  decision  makers.  The  process  encourages 
early  and  frequent  assessments  of  a  system's  status  during 
development,  and  can  result  in  a  significant  reduction  of  test 
time  and  costs  through  comparative  analysis,  data  sharing,  and 
use  of  all  data  sources  for  evaluation.  It  begins  as  early  as 
the  branch  planning  analysis  process  or  battlefield  functional 
mission  area  analysis  for  materiel  systems  and  as  early  as  the 
Project  Management  Plan  (PMP)  process  for  information  systems  and 
continues  through  a  system's  post-deployment  activities.  The  CE 
process  makes  use  of  the  basic  elements  of  T&E  described  in 
paragraph  2-2  to  create  an  integrated  and  continuous  flow  of 
information  on  the  status  of  a  system's  capabilities.  The  CE 
process  is  applicable  to  all  types  of  acquisition  strategies  and 
all  categories  of  acquisition  programs. 

a.  Objectives.  The  objectives  of  CE  are  listed  below. 

(1)  Surface  critical  problems  at  the  earliest  opportunity 
so  they  may  be  addressed  and  resolved  before  they  impact  major 
decisions. 

(2)  Support  the  formulation  of  realistic  system 
requirements  and  specifications  and  ensure  the  system  is 
testable. 

(3)  Provide  for  early  and  frequent  assessment  and 
reporting  of  a  system's  status  during  development. 

(4)  Assure  that  the  system  successfully  transitions  from 
engineering  into  production. 

(5)  Reduce  test  time  and  cost  through  comparison  analyses, 
data  sharing,  and  use  of  all  data  sources  for  evaluation. 

(6)  Monitor  the  corrections  applied  and  assess  the 
adequacy  of  the  corrective  actions  to  be  identified  deficiencies. 

(7)  Provide  assessments  of  system  capabilities  after 
fielding. 
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(8)  Assure  the  system  is  operationally  effective, 
operationally  suitable  and  able  to  satisfy  the  mission  need. 

(9)  Assure  the  system  meets  technical  performance 
requirements . 

b.  Roles.  Continuous  evaluation  throughout  the  life  cycle  of 
a  system  is  performed  by  the  PM/MATDEV,  the  independent 
developmental  evaluator,  and  the  independent  operational  ^ 
evaluator.  Detailed  descriptions  of  the  CE  roles  are  contained 
in  Chapter  3. 

c.  Scope.  Since  CE  applies  to  all  aspects  of  a  system 
throughout  its  life  cycle,  it  has  an  important  role  in  the 
requirements  processs,  the  acquisition  process,  T&E,  and  materiel 
release. 

(1)  CE  in  Support  of  the  Combat  Development  Process 
(Materiel  Systems)  and  the  Information  Mission  Area  Planning 
Process  (Information  Systems) .  Several  primary  documents  are 
generated  by  the  Combat  Developer  (CBTDEV) ,  Functional  Proponent 
(FP) ,  PM,  or  the  MATDEV  which  initiate  the  start  of,  and 
delineate  the  requirements  of  the  materiel  acquisition  process 
(MAP)  or  the  Information  Mission  Area  planning  process.  These 
documents  identify  the  need  for  the  system,  the  functions  it  is 
to  perform,  the  necessary  operational  capabilities,  and  the 
information  which  will  be  used  to  select  the  best  alternative. 
Involvement  of  the  CE  participants  in  the  development  of  these 
documents  is  crucial  to  ensure  the  system  requirements  are 
properly  formed  and  are  addressable  by  T&E.  Figure  2-1  briefly 
discusses  the  purpose  and  content  of  these  documents. 

(a)  Battlefield  Functional  Mission  Area  (BFMA) . 

(b)  Mission  Need  statement  (MNS) . 

(c)  Operational  Requirements  Document  (ORD) . 

(d)  User  Functional  Description  (UFD) . 

(e)  Functional  Description  (FD) . 

(f )  Economic  Analysis  (EA) . 

(g)  Critical  Operational  Issues  and  Criteria  (COIC) . 

(h)  Cost  and  Operational  Effectiveness  Analysis/Cost 
Training  Effectiveness  Analysis  (COEA/CTEA) . 
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(INSERT  FIGURE  2-1  HERE) 

(2)  CE  in  Support  of  the  Materiel  Development  Process 
(Materiel  Systems)  and  the  Information  Mission  Area  Development 
Process  (Information  Systems) .  Program  management  actions, 
organizations,  and  documentation  provide  the  basic  structure  for 
CE.  Testers  and  evaluators  monitor,  review,  and  provide  input  to 
program  management  activities  and  acquisition  program  planning  to 
ensure  that  adequate  resources  are  provided  for  effective  T&E  and 
to  ensure  that  CE  makes  the  maximum  possible  contribution  to 
rapid,  effective,  and  efficient  system  development  and  fielding. 
The  following  Program  Management  elements  are  discussed  in 
Figure  2-2. 

(a)  Acquisition  Strategy  (AS) . 

(b)  Decision  Review  Bodies:  Defense  Acquisition  Board 
(DAB) ,  Army  Systems  Acquisition  Review  Council  (ASARC) ,  Major 
Automated  Information  Systems  Review  Council  (MAISRC) ,  In-Process 
Review  (IPR) . 

(c)  Project  Management  Plan  (PMP) . 

(d)  System  Decision  Paper  (SDP) . 

(e)  Integrated  Program  Summary  (IPS) . 

(f)  Integrated  Program  Assessment  (IPA) 

(g)  Agency  Procurement  Record  (APR) . 

(h)  Request  for  Proposal  (RFP) . 

(i)  Preliminary  and  Critical  Design  Reviews,  and 
Physical  Configuration  Audits  (PDR/CDR/PCA) . 

(INSERT  FIGURE  2-2  HERE) 

(3)  CE  in  Support  of  the  Test  and  Evaluation  Decision 
Process.  The  most  critical  role  played  by  the  CE  process  is  in 
support  of  the  T&E  decision  process.  Test  programs  are 
structured  to  support  evaluation  of  issues  and  system 
requirements.  Planning  for  T&E  is  fully  coordinated  among 
members  of  the  acquisition  team  by  means  of  the  Test  and 
Evaluation  Master  Plan  (TEMP)  and  the  Test  Integration  Working 
Group  (TIWG) .  T&E  is  accomplished  using  a  cycle  of  successive 
actions  and  documents.  For  developmental  T&E,  it  includes  the 
independent  evaluation  or  assessment  plan  (lEP/IAP) ,  the  test 
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design  plan  (TDP) ,  the  detailed  test  plan  (DTP) ,  the 
Developmental  Test  Readiness  Review  (DTRR) ,  Developmental  Test 
Readiness  Statement  (DTRS) ,  the  Test  Reports  (TRs) ,  and  the 
independent  evaluation  reports  and  assessments  reports 
(lERs/IARs).  For  operational  T&E,  it  includes  the  test  and 
evaluation  plans  (TEPs) ,  the  Outline  Test  Plan  (OTP),  the  DTPs, 
the  Operational  Test  Readiness  Review  (OTRR) ,  Operational  Test 
Readiness  Statement  (OTRS) ,  the  TRs,  and  the  lERs.  Each  of  the 
above  items  is  discussed  in  detail  in  Part  One,  Chapter  8,  and 
Parts  Two,  Four,  Five,  Six,  and  Seven. 

(4)  CE  in  Support  of  the  Materiel  Release  Process. 

AR  700-142  provides  a  discussion  on  the  Materiel  Release  process. 
CE  plays  a  vital  role  in  determining  whether  materiel  is  suitable 
for  release.  The  results  of  all  testing,  both  developmental  and 
operational,  must  be  considered  in  all  materiel  release 
decisions.  The  independent  evaluators  must  present  positions  to 
the  MATDEV  relative  to  any  proposed  materiel  release,  and  list 
the  factors  that  could  prevent  a  full  release  of  the  system. 

These  positions  should  address  the  following  issues: 

(a)  The  ability  of  the  system,  when  fielded,  to  meet 
the  contractual  specifications  and  requirements  in  system 
performance,  reliability,  logistic  supportability,  system 
software  design,  and  the  human  factors  engineering  design  of  the 
system . 


(b)  The  degree  to  which  the  system  complies  with  any 
special  directions  or  requirements  issued  by  a  decision  review 
body. 


(c)  The  sufficiency  of  corrections  to  previously 
disclosed  deficiencies,  shortcomings,  and  problem  areas. 

(d)  The  safety  assessment  of  the  system  as  to  its 
operating  and  maintenance  procedures. 
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REQUIREMENTS  DOCUMENTS 


1.  Battlefield  Functional  Mission  Area  concepts  (Materiel 
Systems)  (See  AR  71-9,  AR  70-1).  The  BFMA  is  an  assessment  of 
the  capability  of  a  force  to  perform  within  a  particular 
battlefield  or  functional  area.  The  analysis  is  designed  to 
identify  deficiencies  in  doctrine,  training,  leader 
development,  organizations,  and  materiel  and,  to  identify 
means  of  correcting  these  deficiencies.  The  BFMA  provides  a 
basis  for  applying  advanced  technology  to  future  Army 
operations  and  provides  input  to  the  Battlefield  Modernization 
Development  Plan  (BMDP) ,  the  Battlefield  Development  Plan 
(BDP) ,  the  Mission  Need  Statement  (MNS) ,  and  the  Operational 
Requirements  Document  (ORD) .  Branch  planning  analysis  for 
each  branch  is  conducted  by  the  proponent  school  for  its 
mission  area.  The  TRADOC  Analysis  Center  (TRAC)  provides 
analytic  support  to  each  study.  Statements  of  the  military 
utility  of  developing  systems  and  justification  for  their 
development  are  contained  in  the  results  of  the  BFMA. 

2.  Mission  Need  Statement  (Materiel  Systems  and  Information 
Systems)  (See  DODD  5000.1,  DODI  5000.2,  AR  71-9  and  AR  25-3). 
The  MNS  documents  deficiencies  in  current  capabilities  and 
opportunities  to  provide  new  capabilities  expressed  in  broad 
operational  terms.  Mission  needs  and  resulting  acquisition 
programs  shall  be  based  on  current,  authoritative  threat 
information.  The  MNS  states  the  purpose  of  the  proposed 
system,  where  and  how  it  will  be  used,  the  organizations  that 
will  employ  it,  and  how  it  will  be  integrated  into  the  force 
structure.  It  establishes  readiness  objectives  and  is  the 
basis  for  integrated  logistics  support  (ILS)  planning.  For 
materiel  systems,  prior  to  Milestone  0  (MS  0),  the  MNS  is 
developed  from  the  operational  concept  resulting  from  the 
BFMA.  The  MNS  is  the  basis  for  the  ORD.  The  CBTDEV  prepares 
the  MNS  in  coordination  with  the  MATDEV,  TNGDEV,  and  the 
logistician.  For  information  systems,  prior  to  MS  0,  the  MNS 
is  developed  by  the  FP  and  becomes  the  basis  for  the  User 
Functional  Description  (UFD)  and  FD.  For  both  categories  of 
systems,  it  can  support  the  early  identification  of 
instrumentation  and  test  requirements,  and  the  initial 
determination  of  critical  operational  issues  and  criteria. 

The  statement  is  the  basis  for  early  planning  and  efforts  for 
the  TEMP. 

3.  Requirements  Study  (RS)  (Information  Systems) 

(See  AR  25-1) .  The  needs  contained  in  the  MNS  are  supported 
by  a  requirements  study  commensurate  with  the  size  and 
complexity  of  the  need.  The  RS  should  address  factors  such  as 
the  information  processing  functions  to  be  performed;  the 
problems  that  will  be  solved  by  acquiring  new  or  additional 
equipment;  and  the  nature  of  the  information  to  be  generated. 


Figure  2-1 


3.  Operational  Requirements  Document  (ORD)  (Materiel  Systems) 
(See  DODI  5000.2,  DOD  5000. 2-M,  AR  70-1,  AR  71-9).  The  ORD  is 
the  formal  requirements  document  which  must  be  approved  before 
a  proqram  can  enter  engineerinq  and  manufacturing  development. 
It  is  approved  at  MS  I,  updated  and  expanded  at  MS  II.  It  is 
prepared  primarily  by  the  CBTDEV  in  coordination  with  the 
MATDEV,  TNGDEV,  logistician,  MANPRINT  Planner;  developmental 
and  operational  testers  and  evaluators.  The  ORD  states  the 
threat,  RAM,  technical,  MANPRINT,  logistical,  and  cost 
requirements  to  meet  the  operational  need. 

4.  User  Functional  Description  (UFD)  (Inforaation  Systems, 
and  Materiel  Systems  with  automated  processing  capabilities) 
(See  Part  VII).  The  UFD  provides  the  bridge  between  the  MNS 
and  the  detailed  system  specifications.  It  is  prepared  by  the 
FP  and  defines  the  operational  requirements  in  sufficient 
detail  to  guide  the  software  development.  The  UFD  describes 
implications  of  the  operational  requirements  for  automated 
capabilities  on  the  system's  operational  modes  and  mission 
profile,  proposed  procedures,  interfaces  with  other  systems, 
and  degraded  operations. 

5.  Functional  Description  (FD)  (Information  Systems) 

(See  AR  25-3) .  The  FD  is  prepared  by  the  PM/MATDEV  after  the 
development  of  the  UFD.  The  FD  provides  detailed  requirements 
to  be  used  in  the  development  of  the  system  specifications. 

It  reflects  the  definition  of  the  system  requirements  and 
provides  the  users  with  a  detailed  statement  of  the  required 
operational  capability.  It  also  describes  the  technical 
requirements  needed  of  the  system  to  achieve  the  operational 
requirements  prescribed  in  the  UFD.  For  materiel  systems  with 
a  UFD,  the  ORD,  rather  than  the  FD,  is  the  next  product  in  the 
requirements  generation  process. 

6.  Economic  Analysis  (EA)  (Information  Systems) 

(See  DOD  7920-2. M,  AR  25-1).  The  EA  is  conducted  to  identify 
and  quantify  costs  and  benefits  for  program  alternatives.  It 
considers  such  factors  as  productivity,  availability, 
efficiency,  safety,  quality,  morale,  security,  and 
supportabi 1 ity . 


7.  Critical  Operational  Issues  and  Criteria  (COIC) .  (See 
Part  Three) .  The  primary  purpose  of  COIC  is  to  focus  and 
support  the  MS  III  production  decision.  They  reduce  the 
multitude  of  operational  considerations  to  a  few  operationally 
significant  and  relevant  issues  and  criteria.  COIC  reflect 
the  minimum  operationally  effective  and  suitable  system 
expectation  for  an  affirmative  production  decision;  however, 
they  are  not  to  be  treated  as  automatic  pass/fail  absolutes. 
The  total  operational  system  must  satisfy  the  criteria  (or 
convincing  other  evidence  of  operationally  effective  and 
suitable  system  presented)  for  an  affirmative  production 
decision.  Secondarily,  COIC  focus  and  prioritize  the 
operational  evaluation,  provide  operational  priority  for  the 
acquisition  effort,  and  foster  coordination  among  the 
acquisition  team  members.  COIC  are  not  test  issues  and  can  be 
answered  using  any  suitable  data  source  and  evaluation 
technique.  The  operational  evaluator  must  report  system 
status  against  the  COIC  for  the  production  decision.  The 
total  operational  system  includes  the  materiel,  combat, 
software  and  training  developer  portions.  COIC  apply  to  all 
new  materiel  acquisitions  (ACAT  I  through  IV) ,  class  II 
through  V  IMA  systems,  and  applicable  modifications  to  these 
systems . 

8.  Cost  and  operational  effectiveness  analysis/cost  and 
training  effectiveness  analysis  (COEA/CTEA)  (Materiel  Systems) 
(See  DoDI  5000.2,  DOD  5000. 2-M,  and  AR  71-9).  The  COEA  and 
CTEA  provide  information  on  system  costs  and  operational  and 
training  effectiveness  to  evaluate  the  merits  of  alternatives. 
The  COEA  is  prepared  for  the  MS  I  and  MS  II  decision  reviews 
and  also  update  as  required  for  the  MS  III  decision  review. 

The  MS  I  COEA  is  used  to  narrow  the  list  of  alternatives  to 
the  most  preferred.  The  MS  II  COEA  contains  a  more  detailed 
analysis  to  determine  relative  cost  and  effectiveness  of  each 
alternative  assessed  in  the  demonstration  and  validation 
phase.  The  criteria  and  specifications  which  define  the 
minimum  performance  characteristics  are  to  be  traceable  to  the 
MNS  and  the  ORD. 


Figure  2-1. 
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PROGRAM  MANAGEMENT  ELEMENTS 

1.  Acquisition  strategy  (AS)  (See  AR  70-1  and  AR  25-3).  The 
AS  provides  a  broad,  conceptual  framework  for  the  execution  of 
an  acquisition  program.  It  states  the  concepts  and  objectives 
that  direct  and  control  overall  development,  production,  and 
deployment.  An  AS  is  required  for  all  Army  acquisition 
programs.  The  AS  documents  how  the  acquisition  program  will 
be  tailored  and  identifies  risks  and  plans  to  reduce  or 
eliminate  the  risks.  The  AS  and  the  TEMP  are  developed  in 
parallel  to  ensure  that  the  documents  are  mutually  supporting. 
The  AS,  prepared  by  the  MATDEV  in  coordination  with  the 
acquisition  team,  is  a  living  document  that  matures  throughout 
the  system's  life  cycle.  By  MS  I  for  both  materiel  and 
information  systems,  it  covers  10  functional  areas  including 
T&E,  MANPRINT,  supportability ,  technical  risks,  manufacturing 
and  production,  cost  growth  and  drivers,  human  factors 
engineering  (HFE) ,  safety  and  health,  rationalization, 
standardization,  and  interoperability  (RSI) ,  survivability  and 
endurance,  and  electrical  power  and  environmental  equipment. 
The  AS  is  approved  by  the  appropriate  decision  review  body 
either  as  a  stand-alone  document  or  as  an  element  of  the  IPS 
(for  materiel  systems)  or  the  SDP  (for  information  systems) . 

2.  Decision  review  bodies:  Defense  Acquisition  Board  (DAB), 
Army  Systems  Acquisition  Review  Council  (ASARC) ,  In-process 
Review  Panel  (IPR)  (See  DODD  5000.1,  AR  70-1).  Major 
management  decisions  during  the  acquisition  process  are  made 
at  milestones  by  review  bodies.  The  type  of  review  body 
depends  on  whether  the  acquisition  has  been  categorized  as  an 
ACAT  I,  II,  III,  IV,  or  MAISRC.  For  the  three  program 
management  levels,  the  review  bodies  are  the  DAB,  the  ASARC, 
and  the  IPR  Panel.  For  ACAT  ID  and  OSD  MAISRC  programs,  the 
DAB  reviews  the  critical  issues  and  provides  the  SECDEF  with 
recommendations.  For  ACAT  IC  and  ACAT  II  programs,  the  ASARC 
provides  the  Secretary  of  the  Army  with  recommendations  on  the 
system;  and  similarly  for  Army  MAISRC  programs.  For  nonmajor 
programs,  the  IPR  Panel  provides  recommendations  to  the 
program  executive  officer  (PEO)  or  MATDEV. 


Program  Management  Elements 
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3.  Project  Management  Plan  (PMP)  (Information  Systems) 

(See  AR  25-3).  The  PMP  is  the  primary  document  used  by  the 
PM/MATDEV  to  describe  the  development  of  the  information 
system.  The  PMP  implements  the  PM's  strategy  and  assigns 
responsibility  to  each  participating  agency,  including  testers 
and  evaluators,  and  directs  a  course  of  action  and  method  of 
execution  for  system  development. 

4.  System  Decision  Paper  (SDP)  (Information  Systems) 

(See  AR  25-3).  The  SDP  is  the  primary  management  document  to 
support  an  information  system  through  its  milestone  reviews. 

It  summarizes  the  project,  the  alternatives  considered, 
progress  toward  completion  of  the  project,  and  the  issues.  It 
is  required  for  all  class  II  through  V  information  systems. 

The  SDP  contains  the  AS  and  the  PMP,  and  also  includes  the  EA 
and  TEMP  as  annexes. 

5.  Integrated  Program  Summary  (IPS)  (Materiel  Systems) 

(See  DODI  5000.2,  DOD  5000. 2-M,  AR  70-1).  The  IPS  provides  a 
detailed  summary  of  the  program.  The  IPS  provides  a  succinct 
integrated  pictureof  the  program's  status  for  use  by  the 
decision  review  body.  The  IPS  is  supplemented  by  attachments 
displaying  summaries  of  system  acquisition  costs  and  manpower 
requirements . 

6.  Integrated  Program  Assessment  (IPA)  (Materiel  Systems) 

(See  DODI  5000.2,  AR  70-1)  The  IPA  summarizes  the  results  of 
the  independent  assessments  conducted  by  the  support  staff  and 
decision  review  forums.  The  IPA  is  a  major  issue  oriented 
document.  The  IPA  provides  an  independent  assessment  of  a 
program's  status  and  readiness  to  proceed  into  the  next  phase 
of  the  acquisition  cycle. 


Figure  2-2. 
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7.  Agency  Procurement  Request  (APR)  (Information  Systems) 

(See  AR  25-3) .  The  APR  is  prepared  by  the  PM/MATDEV  or 
contracting  activity  in  order  to  obtain  delegation  of 
procurement  authority  from  the  General  Services  Administration 
for  most  information  systems  exceeding  certain  monetary 
thresholds.  These  thresholds  should  not  be  confused  with 
those  which  define  the  information  system  classes. 

8.  Request, for  Proposal  (RFP)  (See  DODI  5000.2).  The  RFP  is 
developed  by  the  MATDEV  based  on  milestone  decision  reviews 
and  the  AS.  Specifications  in  the  RFPs  are  to  be  traceable  to 
the  MNS,  ORD,  COIC,  and  other  requirement  documentation.  The 
developmental  and  operational  evaluators  are  to  ensure  that 
adequate  numbers  of  test  items  will  be  available  for  testing; 
that  there  are  no  unacceptable  test  limitations  driven  by  the 
RFP;  and  that  provisions  are  made  in  the  RFP  to  provide 
appropriate  contractor  test  data  to  the  independent  evaluators 

9.  Preliminary  and  Critical  Design  Reviews,  and  Physical 
Configuration  Audits  (PDR/CDR/PCA)  (MIL-STD-1521B, 
MIL-STD-2167) .  Technical  reviews  and  audits  provide  a 
valuable  source  of  data  for  developing  test  plans.  The  PDR, 
CDR,  and  PCA  are  periodic  reviews  of  the  detailed  design, 
contractor  testing,  and  operation  and  support  documents  for 
the  system  under  development.  In  addition,  they  provide  data 
useful  in  the  evaluation  of  design  compatibility  between  the 
system  and  other  systems  in  the  field.  A  Physical 
Configuration  Audit  (PCA)  is  a  technical  review  of  a  system 
prototype  to  verify  that  the  end  item  (as  buPt)  conforms  to 
the  technical  documentation  which  defined  the  system.  After 
successful  completion  of  the  PCA,  all  subsequent  system 
changes  are  processed  by  Engineering  Change  Actions. 

Figure  2-2. 
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Chapter  3 

Roles  and  Missions  in  Test  and  Evaluation 
3-1.  Introduction 

a.  Coordination  among  all  participants  of  the  DoD  and  Army 
Test  and  Evaluation  community  is  essential  to  the  success  of  the 
Development  and  Operational  Test  and  Evaluation  processes  in  the 
acquisition  of  Army  systems  and  continuing  evaluation  of  thathose 
systems  after  acquisition.  A  fully  coordinated  and  integrated 
effort  is  necessary  for  timely,  effective,  and  efficient  T&E  that 
is  neither  fragmented  nor  redundant.  The  respective  roles  and^ 
missions  of  the  organizations  within  the  DoD  that  play  a  role  in 
the  T&E  of  Army  systems  are  identified  in  this  chapter. 

b.  The  functional  interactions  among  organizations  of  the 
Army  T&E  community  (figure  3-1)  manage  and  supervise  the  T&E 
process,  accomplish  the  T&E,  and  provide  support  for  T&E.  Many 
of  the  organizations  in  the  T&E  community  perform  multiple 
functions  in  the  T&E  process. 

c.  All  of  the  organizations  in  the  T&E  community  use  and 
review  the  output  of  T&E  to  enhance  the  Materiel  Acquisition 
Process  (MAP)  and  the  Information  Mission  Area  (IMA)  acquisition 
process  and  their  contributions  to  the  T&E  process.  The  T&E 
community  forms  work  groups  to  perform  specific  planning  and 
coordinating  functions  for  T&E  and  to  participate  in  decision 
making  bodies  such  as  the  Defense  Acquisition  Board  (DAB) ,  the 
Army  Systems  Acquisition  Review  Council  (ASARC) ,  Major  Army 
Information  Systems  Acquisition  Review  Council  (MAISRC) ,  and  the 
In  Process  Review  (IPR)  panel.  These  groups  oversee  progress  in 
the  acqusition  processes,  make  recommendations  on  selection  of 
program  alternatives,  and  recommend  whether  programs  should 
proceed  to  the  next  acquisition  phase. 

(insert  figure  3-1) 

Section  I 

Department  Of  Defense  Activities 

3-2.  The  Under  Secretary  of  Defense  for  Acquisition  (USD (A)) 

a.  Exercises  the  responsibilities  and  authorities  in  DoD 
Directive  5134.1,  Under  Secretary  of  Defense  (Acquisition)  and 
DoD  Directive  5000.49,  Defense  Acquisition  Board. 

b.  Establishes  and  publishes  acquisition  management  policies 
and  procedures  that  supplement  and  implement  the  provisions  of 
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DoD  Directive  5000.1,  Defense  Acquisition. 

c.  Prepares  long-range  acquisition  investment  area  analyses 
and,  coordinates  the  funding  of  concept  direction  studies. 

3-3.  Director,  Test  and  Evaluation/Live  Fire  Test  (D,T&E/LFT) 

a.  Serves  as  the  principal  staff  assistant  and  advisor  to  the 
Under  Secretary  of  Defense  for  Acquisition  for  technical 
expertise,  oversight  and  support  to  all  elements  of  the  DoD 
acquisition  system. 

b.  Approves  (in  conjunction  with  DOT&E)  TEMPs  of  all  ACAT  I 
and  OSD  TfieE  oversight  programs. 

c.  Monitors  the  conduct  and  reporting  of  DT&E  for  ACAT  I 
programs  and  other  systems  selected  for  oversight. 

d.  Confirms  that  nuclear  hardness  and  survivability 
objectives  are  achieved  during  DT&E. 

e.  Oversees  the  live  fire  test  program. 

f.  Chairs  the  Defense  T&E  Steering  Group  (DTESG) 

g.  Manages  the  Foreign  Weapon  Evaluation  Program. 

h.  Manages  the  Joint  T&E  Program. 

i.  Establishes  and  maintains  Data  Exchange  Agreements  with 
foreign  nations. 

j .  Plans  and  approves  investments  in  test  and  evaluation 
resources  and  threat  simulators. 

k.  Establishes  and  maintains  DoD  policies  and  instructions 
for  the  Developmental  T&E  (DT&E) . 

l.  Establishes  and  maintains  standard  procedures  for 
conducting  T&E  for  selected  types  of  weapon  systems. 

m.  Manages  the  DoD  Major  Range  and  Test  Facility  Base. 

n.  Oversees  the  Defense  T&E  Professional  Institute. 

o.  Provides  the  independent  OSD  report  of  Live  Fire  Test  and 
Evaluation  to  Congress  on  covered  ACAT  I  and  II  programs  and 
system  modifications  prior  to  the  decision  to  proceed  beyond 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992  (DRAFT) 

LRIP. 


3-4.  Director  of  Operational  Test  and  Evaluation  (DOT&E) 

a.  Reports  directly  to  SECDEF.  The  Office  of  DOT&E  is 
independent  from  the  Under  Secretary  of  Defense  for  Research  and 
Engineering  (USDRE) ,  the  USD(A),  and  all  other  offices  in  DOD 
responsible  for  research  and  development.  The  specific  duties  of 
DOT&E  are  outlined  in  DODD  5141.2. 

b.  Approves  the  acceptability  of  T&E  plans  (TEP) ,  including 
the  adequacy  of  funding  and  resources,  of  operational  testing  and 
evaluation  of  ACAT  I,  OSD  level  MAISRC  IMA,  and  DOT&E  oversight 
ACAT  II,  III,  and  IV  programs.  (See  DODD  5141.2). 

c.  Reports  to  SECDEF  and  Congress  on  the  adequacy  of  T&E  and 
whether  the  results  confirm  the  system’s  operational 
effectiveness  and  suitability  in  support  of  a  final  decision  to 
proceed  with  a  major  program  beyond  low-rate  initial  production 
(LRIP) . 


d.  Prescribes  policies  and  procedures  governing  the  conduct 
of  operational  test  and  evaluation. 

e.  Provides  independent  assessments  and  reports  as  required 
by  current  statutes. 

f.  Issues  DOD  Instructions  to  implement  policies  for  OT&E. 

g.  Obtains  reports,  information,  advice,  and  assistance  as 
necessary  to  carry  out  assigned  functions.  DOT&E  has  access  to 
all  records  and  authenticated  data  in  the  DOD,  including  all  DOD 
components . 

h.  Approves  (in  conjunction  with  D,T&E/LFT)  Test  and 
Evaluation  Master  Plans  (TEMPs)  for  acquisition  of  ACAT  I,  OSD 
level  MAISRC  IMA,  and  OSD  designated  T&E  oversight  programs  (ACAT 
II,  III,  and  IV  systems) .  Written  approval  is  required  before 
operational  testing  may  begin. 

i.  Observes  preparation  and  conduct  of  OT&E. 

j.  Prepares  annual  OT&E  report  to  Congress  (including  Army 
OT&E)  . 


k.  Serves  as  advisor  to  the  Defense  T&E  Steering  Group 
(DTESG) . 
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3-5.  Service  Component  Operational  Test  Activities  (OTA) 

The  head  of  each  Military  Department  and,  as  appropriate.  Defense 
Agency  have  established  an  independent  operational  test  and 
evaluation  activity  (See  figure  3-2  and  DODD  5000.1).  These 
activities: 

a.  Are  separate  and  independent  from  the  materiel-developing 
and  procuring  agency,  and  the  using  agency. 

b.  Are  responsible  for  planning  and  conducting  operational 
tests,  reporting  results,  and  providing  evaluations  of  each 
tested  system's  operational  effectiveness  and  suitability. 

c.  Report  directly  to  the  head  of  the  DOD  Component,  except 
that  the  Secretary  of  a  Military  Department  may  delegate 
responsibility  for  supervising  this  activity  to  the  Service  Chief 
concerned. 

■3-6.  Defense  Information  Systems  Agency  (DISA) 

a.  DISA  is  responsible  for  operational  test  and  evaluation  of 
strategic  IMA  systems  for  which  no  Lead  Military  Department  or 
equivalent  has  been  assigned. 

b.  The  Joint  Tactical  Communications  Agency's  (JTC3A) ,  Joint 
Interoperability  Test  Center  (JITC)  is  the  DISA's  responsible 
operational  test  agency  (OTA) .  JITC  will  conduct  operatipnal 
test  and  evaluation  in  a  mission  and  threat  environment  a^ 
operationally  realistic  as  possible,  in  accordance  with  DoD 
Directive  5000.2.  In  this  capacity,  the  Director,  JTC3A  is* the 
independent  test -agency  for  all  DISA  acquired  C3I  systems. 

3-7.  Defense  Test  and  Evaluation  Steering  Group  (DTESG) 

DTESG  advises  on  the  DoD  corporate  direction  and  guidance  for  the 
management  of  defense  T&E  resources,  including  those  for 
operational  T&E.  It  consists  of  senior  members  and  advisors  from 
D,T&E/LFT,  DOT&E,  Army,  Navy,  Air  Force,  Defense  Nuclear  Agency, 
Strategic  Defense  Initiative  Organization,  from  the  Science  and 
Technology  Community,  and  the  Chairman  of  the  Joint  Commanders 
Group  for  Test  and  Evaluation  (JCG(T&E)). 

a.  Formulates  DoD  corporate  planning  and  programming  guidance 
for  T&E  resources,  including  threat  simulators,  and  simulations. 

b.  Conducts  reviews  of  DoD  T&E  business  management  practices; 
the  execution  of  directed  T&E  investment  programs,  projected  and 
actual  workloads,  and  test  program  utilization  of  test 
facilities. 
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c.  Formulates  initiatives  that  streamline  and  improve  the 
productivity  and  effectiveness  of  test  and  evaluation  resource 
management  and  the  conduct  of  test  and  evaluation. 

3-8.  Joint  Commanders  Group  for  Test  and  Evaluation  (JCG(T&E)) 
The  JCG(T&E)  advises  the  Joint  Logistics  Commanders  on  issues  and 
opportunities  of  common  interest  to  avoid  duplication  of  efforts 
in  all  matters  of  operational  and  developmental  test  and 
evaluation  operations  and  resources.  Also  manage  the  DoD-wide 
T&E  Community  Network  (TECNET) . 

3-9.  Multi-Service  Test  Investment  Review  Committee  (MSTIRC) 

The  MSTIRC  advises  the  JCG(T&E)  on  operational  and  developmental 
T&E  investments  (including  threat  simulators,  and  modeling  and 
simulations)  needed  by  DoD,  and  recommends  strategies  to  obtain 
them  while  avoiding  duplication  and  promoting  inter-service 
reliance. 

3-10.  Defense  Modeling  and  Simulation  Organization  (DMSO) 

The  DMSO  promulgates  policies  to  facilitate  DoD-wide  applications 
of  modeling  and  simulations,  including  for  test  and  evaluation. 
Also,  implements  programs  for  joint  Service  modeling  and 
simulation  improvements  and  investments. 

Section  II 

Headquarters,  Department  Of  Army  Activities 

3-11.  Army  Acquisition  Executive  (AAE) 

The  AAE  has: 

a.  Authority,  responsibility,  and  accountability  for  all 
acquisition  functions  and  programs  within  the  Army  as  provided 
for  in  DODD  5000.1  and,  for  enforcing  the  procedures  established 
by  the  Under  Secretary  of  Defense  for  Acquisition. 

b.  Authority  to  review  and  provide  assessment  of  any  changes 
reported  in  individual  major  defense  acquisition  programs,  the 
significance  of  problems  reported  by  the  Program  Manager,  the 
Program  Manager's  proposed  action  plans,  and  the  level  of  risk 
associated  with  such  plans. 

c.  Serves  as  principal  advisor  in  the  Army  on  all  matters 
relating  to  acquisition  management  to  include  resource  allocation 
decisions. 

d.  Participates  in  the  selection  and  evaluation  of  Program 
Executive  Officers  (PEOs)  and  Program  Managers  (PMs)  for  major 
defense  acquisition  programs. 
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e.  For  ACAT  ID  programs,  approves  the  Army  position  prior  to 
the  DAB.  For  ACATs  IC  and  II,  serves  as  the  MDA.  For  ACAT  III 
and  IV,  assigns  the  MDA. 

3-12.  Deputy  Under  Secretary  of  the  Army  for  Operations  Research 

(DUSA(OR))  .  ^  ^  .  4.U 

The  DUSA(OR)  is  an  element  of  the  Army  Secretariat  and  is  tne 

principal  advisor  to  the  Secretary  of  the  Army  for  matters 

concerning  Army  T&E. 

a.  Establishes,  reviews,  and  supervises  Army  T&E  policy  and 
procedures . 

b.  Provides  oversight  of  all  Army  T&E  associated  with  the 
system  research,  development  and  acguisition  as  well  as  those 
associated  with  doctrine,  training,  force  design,  leader 
development,  and  materiel  requirements  programs. 

c.  Approves  all  T&E  documents  requiring  OSD  review: 

(1)  TEMPS  and  TEPs  for  HQDA  prior  to  submission  to  OSD. 

(2)  Monitors  the  T&E  program  for  the  Army. 

d.  Provides  staff  management  of  all  user  test  programs  of 
interest  to  the  Office  of  the  Secretary  of  the  Army  (OSA) . 

3-13.  Assistant  Secretary  of  the  Army  for  Research,  Development, 
and  Acquisition  (ASARDA) 

The  ASA(RDA): 

a.  Serves  as  a  member  of  the  Test  Schedule  and  Review 
Committee  (TSARC) .  (See  AR  15-38.) 

b.  Plans,  programs  and  budgets  RDTE  and  Other  Procurement, 
Army  (OPA)  funds  for  test  and  evaluation,  in  support  of  test 
programs . 

c.  Supports  test  requirements,  scheduling,  and  funding  to 
provide  adequate  items  or  systems  for  test. 

3-14.  The  Deputy  Chief  of  Staff  for  Operations  and  Plans 
(DCSOPS) 

The  DCSOPS: 

a.  Plans,  programs,  and  budgets  Operations  and  Maintenance, 
Army  (OMA)  T&E  funds  for  OPTEC  and  for  the  conduct  of  FDTE  and 
FOTE. 
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b.  Participates  in  the  TSARC  process.  (See  AR  15-38.) 

c.  Reviews,  coordinates,  and  approves  requirements  and  COIC 
for  ACAT  I,  II,  OSD  T&E  oversight  ACAT  III  and  IV  programs,  and 
theater /tactical  MAISRC  information  systems. 

d.  Serves  as  HQDA  point  of  contact  and  oversight  for  OSD 
chartered  Joint  Test  and  Evaluation  (JT&E) .  Managing, 
soliciting,  and  coordinating  Army  participation  in  JT&E  tests. 
Providing  Army  members  to  the  JT&E  Planning  Committee  and  JT&E 
Senior  Advisory  Council.  Providing  Army  liaison  to  OSD  on  JT&E 
issues. 


e.  Approves  requirements  for  Instrumentation,  Targets,  and 
Threat  Simulators  (ITTS)  as  chartered  by  the  ITTS  General  Officer 
Steering  Committee. 

f.  Manages  and  monitors  funding  of  Army  Threat  Simulator 
Program  (ATSP) ,  Army  Targets  Program  (ATP) ,  Army  Instrumentation 
Program  (AIP) ,  and  interfaces  with  PM  ITTS, 

g.  Establishes  and  chairs  Test  and  Analysis  Integration 
Working  Groups  (TAIGs)  to  execute  W&A  of  models  and  simulations 
in  support  of  test  and  evaluation. 

3-15.  The  Deputy  Chief  of  Staff  for  Logistics  (DCSLOG) 

The  DCSLOG  has  primary  Army  General  Staff  responsibility  for 
integrated  logistics  support  (ILS)  and  related  T&E  policy  to 
include  input  to  program  management  documents.  (See  AR  750-1  and 
AR  700-127.)  The  DCSLOG  participates  in  the  TSARC  process  as 
required.  The  DCSLOG  is  the  Functional  Proponent  (FP)  for  some 
sustaining  base  IMA  systems.  As  the  FP,  DCSLOG  develops, 
coordinates,  and  obtains  approval  of  COIC,  signs  as  the  user 
representative  on  TEMP  cover  page  before  submission  to  DUSA(OR) 
for  approval,  provides  Test  Support  Packages  (TSP)  for 
operational  test  and  otherwise  is  the  user  representative. 

(NOTE:  When  an  IMA  system  is  both  sustaining  base  and 

Theater/Tactical,  the  combat  developer  accomplishes  these  COIC 
and  TEMP  functions) . 

3-16.  The  Deputy  Chief  of  Staff  for  Personnel  (DCSPER) 

The  DCSPER  has  primary  Army  General  Staff  responsibility  for: 

a.  Manpower,  Personnel,  Training,  Human  Factors  Engineering, 
Health  Hazards  and  System  Safety  (MANPRINT) . 

b.  Ensuring  that  MANPRINT  T&E  concerns  are  addressed  in 
appropriate  testing  and  T&E  documents. 
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c.  Participating  in  the  TSARC  process.  (See  AR  15-38.) 

3-17.  The  Deputy  Chief  of  Staff  for  Intelligence  (DCSINT) 

DCSINT  has  responsibility  for: 

a.  Providing  guidance  on  representation  of  threat  in  testing. 

b.  Establishing  threat  policy  and  procedures  and  approval  of 
the  threat  to^  be  used  for  test  and  evaluation. 

3-18.  The  Director  of  Information  Systems  for  Command,  Control, 
Communications  and  Computers  (DISC4) 

The  DISC4: 

a.  Manages  IMA  activities  in  support  of  the  Army  Acquisition 
Executive  (AAE) ,  to  include  T&E  IMA  life  cycle  management. 

b.  Plans,  programs  and  budgets  Operations  and  Maintenance, 
Army  (OMA)  funds  for  fixed  and  recurring  costs  for  T&E  conducted 
by  U.S.  Army  Information  Systems  Command  (USAISC) . 

c.  Approves  COIC  for  classes  2  through  5  startegic  and 
sustaining  base  IMA  systems.  Establishes  Army  procedural 
guidance  for  developement,  coordination,  and  approval  of  COIC  for 
these  systems.  Assists  the  Deputy  Chief  of  Staff  for  Operations 
and  Plans  in  reviewing,  coordinating,  and  approving  requirements 
and  COIC  for  theater /tactical  information  systems. 

d.  Assigns  operational  test  and  evaluation  (OT&E) 
responsibilities  for  non-MAISRC  level  information  systems  through 
the  IMA  Architectural  Control  Committee. 

e.  Designates  the  OT&E  responsibilities  for  strategic 
information  systems  when  the  Army  is  assigned  as  the  lead 
jniiitary  department.  Designation  will  be  in  coordination  with 
the  Defense  Communications  Agency  (DCA)  and  OPTEC. 

f.  Assists  the  DUSA(OR)  in  developing  IMA-related  test 
policy. 

3-19.  The  Surgeon  General  (TSG) 

The  TSG: 

a.  Provides  support  to  testers  and  evaluators  concerning 
health  hazards. 

ji^uthor izes  the  use  of  humans  as  volunteers  for  test  and 
evaluation. 
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c.  Performs  health  hazard  assessments. 

d.  Reviews  or  participates  in  preparation  of  safety  releases 
as  required. 

e.  Serves  as  a  program  sponsor  for  tests  of  medical  materiel. 

f.  Participates  in  the  TSARC  as  required.  (See  AR  15-38.) 

3-20.  The  Chief  of  Engineers  (COE) 

The  COE: 

a.  Provides  support  to  program  sponsors  in  development  of 
materiel  for  operation  in  extreme  climatic  conditions. 

b.  Provides  policy,  guidance,  and  support  of  tests  and 

evaluations  with  regard  to  the  subject  of  environmental  effects 
on  Army  materiel  and  operations.  ^ 

c.  Executes  the  test  aspects  of  those  commercial  items  of 
equipment  procured  for  use  in  engineer  maintenance  and  supply 
activities. 

d.  Reviews  digital  terrain  data  for  accurate  representation 
in  demonstrations  and  tests. 

e.  Acts  as  program  sponsor  for  COE  acquisition  programs. 

f.  Participates  in  the  TSARC  as  required.  (See  AR  15-38.) 

g.  COE  (Washington,  DC, .and  Fort  Belyoir,  VA)  is  both  a 
proponent  for  engineer  materiel  acquisition  and  a  designated  OT 
activity.  COE  executes  tests  of  commercial  engineer  equipment 
and  monitors  the  OT  and  CE  programs  for  engineering  impacts. 
Testing  of  military  developmental  engineering  equipment  is 
accomplished  by  OPTEC. 

3-21.  Director,  Test  and  Evaluation  Management  Agency  (TEMA) 
TEMA,  Office  of  the  Chief  of  Staff,  Army,  reports  operationally 
to  the  DUSA(OR),  and  is  responsible  to: 

a.  Develop  and  monitor  Army  developmental  and  operational 
test  policy. 

b.  Coordinate  all  test  and  evaluation  policy  and  resource 
actions  with  Office,  Assistant  Secretary  of  the  Army  (Research, 
Development  and  Acquisition),  (OASA(RDA)),  other  HQDA  agencies, 
OSD,  Chief  of  Naval  Operations,  (CNO) ,  Headquarters  US  Air  Force 
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(HQ  USAF) ,  U.S.  Army  Operational  Test  and  Evaluation  Command 
(USAOPTEC) ,  U.S.  Army  Materiel  Command  (USAAMC) ,  U.S.  Army 
Training  and  Doctrine  Command  (USATRADOC) ,  U.S.  Army  Strategic 
Defense  Command  (USASDC) ,  U.S.  Army  Information  Systems  Command 
(USAISC) ,  U.S.  Army  Health  Services  Command  (USAHSC)  and  U.S. 

Army  Intelligence  and  Security  Command  (INSCOM) . 

c.  Manage  the  HQDA  staffing  and  approval  process  for  TEMPs 
requiring  HQDA  and  OSD  approval. 

d.  Oversee  development,  update  and  accreditation  of  test  and 
evaluation  related  models  and  simulations. 

e.  Coordinate  and  facilitate  communications  with  OSD  on  test 
and  evaluation  matters. 

f.  Develop  and  monitor  Army  Major  Range  and  Test  Facility 
Base  MRTFB)  management  and  funding  policy. 

g.  Coordinate  and  oversee  T&E  funding  for  investment 
Research,  Development,  Test  and  Evaluation  (RDTE)  and  Other 
Procurement,  Army  (OPA)  accounts  and  user  test  support. 

h.  Oversee  development  of  T&E  personnel  strategy  plans  for 
identifying  and  training  individuals. 

i.  Oversee  Army  Joint  T&E  and  multiservice  test  programs. 

j .  Serves  as  Army  representative  to  the  Department  of  Defense 
(DOD)  Executive  Committee  for  threat  simulators  and  targets. 

Section  III 

U.S.  Army  Operational  Test  and  Evaluation  Command  (OPTEC) 

3-22.  OPTEC  General 

DODD  5000.1  requires  that  the  DOD  Components  maintain  OTA's  that 
are  separate  from  the  development  and  using  commands  and  report 
directly  to  the  heads  of  the  respective  services.  The  Army's 
primary  independent  operational  T&E  activity  is  OPTEC.  As  stated 
in  AR  73-1,  the  Commanding  General,  OPTEC,  supports  the  MAP,  IMA 
acquisition  process,  and  force  development  process  through 
overall  management  and  support  of  the  Army's  OT  and  CE  programs. 
OPTEC 's  key  contribution  to  the  MAP  and  IMA  acquisition  process 
is  the  testing  and  independent  evaluation  of  the  actual  and 
projected  effectiveness  and  suitability  of  developing  materiel 
and  IMA  systems.  The  OPTEC  mission  is  stated  in  detail  in  AR 
10-88,  4  April  1990. 
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a.  Reporting  channels.  OPTEC  is  a  field  operating  agency  of 
CSA.  OPTEC  reports  through  VCSA  to  CSA;  coordinates  closely  with 
the  MATDEV  and  CBTDEV  communities;  and  ensures  that  the  HQDA, 
MATDEV,  CBTDEV,  and  logisticians  are  informed  of  the  operational 
condition  of  each  system  in  the  acquisition  process.  The 
Commanding  General,  OPTEC,  is  the  chairman  of  the  Test  Schedule 
and  Review  Committee  (TSARC)  and  a  member  of  ASARC  or  the  IPR 
panel  for  each  system.  OPTEC  may  communicate  and  coordinate 
directly  with  any  DOD  or  U.S.  Government  activity  to  obtain 
information  or  assistance  in  support  of  its  mission. 

b.  Continuous  Evaluation  (CE)  functions.  A  major  aspect  of 
OPTEC 's  mission  is  conduct  of  the  Army's  CE  program  (see  chap  3). 
Throughout  the  MAP,  CE  uses  data  from  all  available  sources  to 
provide  decision  makers  with  evaluations  of  current  and  projected 
system  status.  Sources  of  data  for  the  CE  process  include 
contractor  tests,  DT,  OT,  FDTE,  modeling,  simulations,  analyses, 
prior  system-related  T&E,  combat,  and  any  other  available 
information.  CE  functions  include: 

(1)  Support  the  development  of  CE  policy  and  methodology. 

(2)  Evaluating  programs  designated  for  CE  continuously  from 
program  initiation  through  postfielding  deployment,  as  required. 

(3)  Performing  an  abbreviated  evaluation  on  non-major  ACAT 
III  and  IV  programs,  except  those  designated  for  full-evaluation. 

(4)  Periodically  evaluating  for  the  materiel  acquisition 
decision  makers  the  effectiveness  of  the  systems  undergoing  CE, 
describing  corrective  actions  and  identifying  tests  that  will 
verify  operational  effectiveness  and  suitability  after  corrective 
actions  have  been  completed. 

(5)  Evaluating  current  and  projected  effectiveness  of 
supporting  program  elements. 

c.  Operational  testing  program  functions.  OPTEC  is  the 
manager  of  the  Army's  Operational  Test  (OT)  program  (AR  73-1). 

OT  includes  FDTE,  CEP,  CT,  EUTE,  LUT,  lOTE,  FOTE,  onsite 
operational  tests,  joint  operational  tests  and  evaluations  (JOTE) 
and,  multi-service  operational  tests  and  evaluations  (MOTE)  (See 
chapter  12) .  OPTEC  manages  OT  by: 

(1)  Developing  and  promulgating  OT&E  policy  and  methodology. 

(2)  Preparing  the  OT&E  portion  of  the  TEMP  for  all  OPTEC 
evaluated  systems,  and  performing  overall  quality  assurance  of 
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content  and  compliance  with  format  of  the  entire  TEMP. 

(3)  Informing  the  acquisition  decision  maker  when  acquisition 
strategies  preclude  adequate  OT&E  from  being  accomplished. 

(4)  Ensuring  that  OT&E  conducted  by  other  Army  OT&E 
activities  is  effectively  planned,  conducted,  and  reported. 

(5)  Developing  and  staffing  the  Army  position  on  OT&E 
matters,  plans,  and  reports  for  submission  to  OSD  and  Congress. 

(6)  Recommending  the  conduct  of  FDTE  to  the  TRADOC  when  such 
testing  is  needed  to  support  the  development  and  evaluation  of  of 
the  readiness  of  TRADOC  doctrine,  training,  leadership 
development,  and  orgranizational  products  for  the  system  prior 
Initial  Operational  Test  (lOT)  for  ACAT  I,  II  and  other 
acquisition  programs. 

(7)  Publishing  the  Five-Year  Test  Program  (FYTP)  after  the 
TSARC  coordination  and  DCSOPS  approval. 

(8)  Supporting  OSD-directed  Joint  Operational  Testing  and 
Evaluation  (JOTE)  and  MOTE  by  providing  the  Army  point  of 
contact,  developing  and  staffing  the  Army  position  on  joint  test 
documentation,  and  coordinating  Army  resource  support,  as 
required. 

(9)  Managing  the  Army  User  Testing  Instrumentation  Program  in 
coordination  with  TSARC. 

(10)  Planning,  programming,  and  budgeting  to  accomplish 
OPTEC's  OTE  management  functions. 

d.  Responsible  for  maintaining  a  long  range  plan  for 
operational  T&E  resource  requirements. 

e.  OPTEC  is  the  Army's  primary  independent  operational  tester 
and  evaluator,  with  the  exception  of  those  specialized  systems 
assigned  to  the  U.S.  Army  Health  Services  Command  (USAHSC)  for 
medical  materiel,  U.S.  Army  Intelligence  and  Security  Command 
(USAINSCOM) ,  and  Corps  of  Engineers  (COE) .  OPTEC  manages  the 
Operational  Test  Program  and  operational  testing  performed  by 
other  organizations,  and  coordinates  the  overall  OT&E  process. 

3-23.  OPTEC  Operational  Evaluation  Command  (OEC) 

The  Operational  Evaluation  Command  is  under  OPTEC  and  supports 
the  Army  Acquisition  process  by  managing  OPTEC's  Continuous 
Evaluation  Program  to  ensure  timely,  complete,  and  independent 
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operational  evaluations. 

a.  Conducts  full  independent  operational  evaluations  for: 

(1)  ACAT  I  and  II  programs. 

(2)  ACAT  III  and  IV  programs  designated  for  DOT&E  oversight 
or  selected  for  full  evaluation  by  HQDA  or  Commander,  OPTEC. 

(3)  ACAT  III  and  IV  programs  which  may  require  evaluation 
based  upon  program  cost,  system  complexity,  procurement  risk, 
doctrinal  change,  relationships  with  other  systems,  or 
requirements  for  data  beyond  operational  test  as  determined  by 
Commander ,  OPTEC . 

(4)  Selected  IMA  MAISRC  systems. 

b.  Conducts  abbreviated  operational  evaluation  for: 

(1)  All  other  ACAT  III  and  IV  programs  supported  by  a  TEXCOM 
Test  and  Evaluation  Report  (TER)  on  results  of  operational  tests 
or  by  sources  such  as  simulations,  modeling,  market  surveys, 
combat,  training  exercises,  or  developmental  testing. 

(2)  IMA  MAISRC  systems  and  selected  non-MAISRC  IMA  systems 
where  evaluation  responsibility  has  been  delegated  to  another 
Army  activity. 

c.  Performs  analyses  on  authenticated  data  (level  3  data) 
that  has  been  approved  by  the  Data  Authentication  Groups  (DAG) , 
and  generates  Level  4,  5,  6,  and  7  data  in  support  of  operational 
and  continuous  evaluations. 

d.  Reviews  and  coordinates  on  Mission  Need  Statements  (MNS) , 
and  Operational  Requirements  Documents  (ORDs) . 

e.  Works  with  CBTDEV  and  FP  during  their  development  of 
Critical  Operational  Issues  and  Criteria  (COIC) ,  and  develops  and 
approves  Additional  Operational  Issues  and  Criteria  (AOIC) . 

f.  Develops  evaluation  concepts,  including  identifying 
measures  of  effectiveness  (MOEs)  -  levels  5,6  or  7  data;  measures 
of  performance  (MOPs)  -  level  4  data;  data  elements  required;  and 
performs  design  of  experiments  or  tests  for  obtaining  required 
data  to  perform  evaluations. 

g.  Responsible  for  the  operational  model-test -model  program, 
including  developing  and  executing  Verification,  Validation,  and 
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Accreditation  Plans  for  the  use  of  modeling  and  simulation  in 
OT&E.  Recommends  accreditation  of  models  and  simulations  to  Hq 
OPTEC. 


h.  Participates  in  Test  Integration  Working  Groups  (TIWG) 

(See  Section  3-54) . 

i.  Participates  in  Test  and  Analysis  Integration  Groups 
(TAIGs)  see  Section  3-51. 

j.  Chairs  Operational  RAM  Scoring  Conferences  for  operational 
evaluations,  and  the  RAM  Assessment  Conference  -  see  Section  3- 
57.  Coordinates  on  Failure  Definition/Scoring  Criteria. 

k.  Provides  inputs  to  the  PM  for  exit  criteria  for  Milestones 
I  and  II,  LRIP  release,  as  well  as  for  other  specific  operational 
T&E  events. 

l.  Prepare  inputs  for  Operational  Test  Plan  for  resources  to 
support  the  TSARC. 

m.  Establishes  and  reports  on  progress  for  meeting  entrance 
criteria  for  entering  an  OT&E  or  CE  event  at  Operational  Test 
Readiness  Reviews. 

n.  Publishes  operational  assessments,  independent  evaluation 
reports,  and  operational  test  and  evaluation  reports. 

o.  Performs  quality  control  for  DA  on  overall  TEMPs  as  well 
as  prepares  Part  4. 

p.  Prepares  Chapters  1  and  2  of  the  TEP,  and  integrates 
overall  document. 

q.  Chairs  System  Task  Forces  to  coordinate  all  test, 
evaluation  and  analysis  disciplines  (eg,  MANPRINT,  RAM,  software, 
and  performance)  required  to  support  OT&E  or  CE  for  a  particular 
system  or  program. 

3-24.  OPTEC  Test  and  Experimentation  Command  (TEXCOM) 

TEXCOM  is  under  OPTEC  and  supports  the  Army  materiel  acquisition 
and  force  development  processes  by  executing  the  user  testing 
program  by  conducting  operational  testing  to  support  CE  and  force 
development.  All  TEXCOM  elements  are  responsible  for  continuous 
improvement  of  processes  for  the  purpose  of  optimizing  resources 
and  improving  products.  TEXCOM  is  responsible  for  the  following: 

a.  Plans,  conducts,  collects  data  and  reports  the  results  of 


3-14 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992 


(DRAFT) 


operational  tests,  special  studies,  commercial  equipment 
evaluations,  customer  tests,  and  field  experiments.  Remains 
prepared  to  conduct  quick  response  tests  involvinq  safety  and 
improvements  to  operational  readiness  of  fielded  units.  Types  of 
tests  conducted  include,  but  are  not  limited  to  EUTE,  CT,  LUT, 
FDTE,  CEP,  lOTE,  and  FOTE. 

b.  Conducts  comprehensive  programs  for  developing  and 
improving  test  methodology,  test  instrumentation,  test 
facilities, , and  ranges  to  support  the  testing  mission.  Develops 
and  monitors  contract  specifications  and  technical  specifications 
for  procurement  packages. 

c.  Provides  data  collection  training  and  maintains  individual 
soldier  skills  to  Army  standards. 

d.  Provides  direction  to  subordinate  elements  in  the 
management  of  test  and  experimentation  programs.  Supervises  test 
team  formulation  and  manages  formatting,  coordination,  and 
processing  for  TSARC  approval  of  resource  requirements  and 
documentation  for  each  test  assigned. 

e.  Reviews  and  comments  on  test  aspects  of  test-related 
documents  to  include  mission  need  statements,  operational 
requirements  documents,  test  and  evaluation  plans,  resume  sheets, 
and  test  support  packages)  that  are  received  from  proponent 
schools,  independent  evaluators,  and  materiel  developers. 

f.  Maintains  close  and  continuous  coordination  with  combat 
developers  in  regard  to  the  development  and  testing  of  tactics, 
doctrine,  training,  organization  and  material. 

g.  Consults  and  coordinates  with  other  DOD  armed  services, 
allied  nations  armed  services,  industry,  and  test  sponsors  on  the 
test  mission  programs  and  scientific  and  technical  aspects  of 
materiel  testing  and  experimentation. 

h.  Performs  special  projects  and  staff  actions  and  supports 
the  operations  of  HQDA,  OPTEC,  and  DOD  in  test-related  areas. 
Provides  membership  or  representation  on  boards,  panels, 
committees,  councils,  working  groups,  symposia,  and  conferences 
as  approved  by  the  Commander,  TEXCOM. 

i.  Provides  advice  to  proponent  agencies  and  materiel 
developers  during  the  development  of  equipment  which  is  either 
used  by  or  provides  support  to  Army  units. 

j.  Coordinates  and  assists  in  submission  of  environmental 
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impact  statements  for  tests.  Responsible  for  implementation  of 
Hazardous  Waste  Management  Plan  and  protection  of  historical 
properties. 

k.  Participates  in  Test  Integration  Working  Groups  (TWIGS)  as 
operational  tester. 

l.  Conducts  tests  for  TRADOC  schools  and  centers. 

m.  Ensures  that  combined  arms  skills,  doctrine,  and  training 
are  being  integrated  into  operational  testing. 

n.  TEXCOM  Experimentation  Center  (TEC) .  A  subordinate  center 
of  TEXCOM,  located  at  Fort  Hunter  Liggett,  CA,  conducts 
scientific,  highly  instrumented  field  experimentation  and 
collects  high-resolution,  accurate  data  in  an  operational 
environment . 

o.  Participates  as  a  member  of  the  System  Task  Forces. 

3-25.  OPTEC  Operational  Threat  Support  Activity  (OTSA) 

OTSA  a  subordinate  element  of  OPTEC,  is  located  at  Fort  Bliss, 
Texas  and  assists  and  advises  the  Commander,  OPTEC  in  the 
fulfillment  of  the  OPTEC  assigned  responsibility  for  Army  Threat 
Simulator  (ATS)  Program  actions.  Operates  and  maintains 
operating  replica  simulators  and  actual  threat  systems  and 
ensures  that  realistic  threat  environments  are  used  in  support  of 
free-play,  force-on-force,  real-time  casualty  assessment  testing 
and  training.  Responsible  for  continuous  improvement  of 
processes  for  the  purpose  of  optimizing  resources  and  improving 
products . 

3-26.  OPTEC  Test  and  Evaluation  Coordination  Offices  (TECO) 

TECOs  are  subordinate  elements  of  OPTEC  and  provide  on-site 
coordination  between  OPTEC  and  the  TRADOC  Proponent  Center, 
Provide  operational  test  and  evaluation  (T&E)  expertise  to  the 
TRADOC  proponent  activity.  The  TECO  is  responsible  for 
continuous  improvement  of  processes  for  the  purpose  of  optimizing 
resources  and  improving  products.  TECOs  are  located  at  Fort 
Rucker,  AL,  Fort  Knox,  KY,  Fort  Leonard  Wood,  MO,  Fort  Banning, 
GA,  Fort  Lee,  VA  and  Fort  Gordon,  GA. 

3-27.  Other  Operational  Testers 

In  general,  TEXCOM  accomplishes  operational  testing  for  ACAT  I, 
II,  III  and  IV  and  selected  IMA  programs  for  the  Army.  Other 
agencies  conduct  specialized  operational  testing  in  the  areas  of 
communications,  intelligence  (INSCOM) ,  medical  materiel  (USAHSC) , 
new  technology,  and  commercial  engineer  equipment  (CEE) .  For 
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assicfned  systems  the  designated  operational  testers. 

a.  Participate  in  development  of  issues  and  criteria. 

b.  Identify  CBTDEV  provided  issues  to  be  addressed  in  OT&E. 

c.  Plan  program  and  budget,  conduct,  and  report  on  OT&E  to 
support  the  materiel  acquisition  decision  review  process. 

d.  Assist  in  preparation  of  the  OT&E  portion  of  the  TEMP. 

e.  Prepare  test  plans  and  reports  and  coordinate  with  OPTEC 
and  TEXCOM  as  appropriate. 

Section  IV 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC) 

3-28.  TRQDOC  General 

TRADOC  is  the  Army's  principal  doctrine  developer,  combat 
developer,  training  developer,  and  trainer.  As  such  TRADOC: 

a.  Establishes  the  operational  concepts  and  doctrine  for 
field  Army  operations  on  the  future  battlefields. 

b.  Analyzes  and  projects  the  future  threat  in  coordination 
with  other  appropriate  Army  .and  DOD  organizations. 

c.  Defines  Army  organizational,  leader  development,  training, 
and  materiel  operational  requirements. 

d.  Represents  the  user  during  all  aspects  of  requirement 
definition,  system  acquisition,  and  force  development. 

e.  Conducts  individual  and  collective  training  as  well  as 
leader  development  programs  for  soldiers. 

3-29.  HQ  TRADOC 

a.  Plans,  programs,  budgets,  and  executes  the  Concept 
Evaluation  Program  (CEP)  and  the  Force  Development  Test  and 
Experimentation  (FDTE)  program  in  support  of  above. 

b.  Conducts  Concept  Evaluation  Schedule  and  Review  Committee 
(CEPSARC)  to  coordinate,  prioritize,  and  approve  projects  for 
execution  in  the  CEP  Program. 

c.  Represents  TRADOC  during  TSARC  processes  (includes: 
impacting  test  resource  requirements  on  TRADOC,  approving  and 
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prioritizing  the  proposed  FDTE  projects  for  execution, 
participating  and  concurring  in  the  prioritization  of  other 
operational  tests,  and  participating  in  resolution  of  other 
issues) . 

d.  Provides  procedural  guidance  for  execution  of  the  TRADOC 
T&E  function. 

e.  Approves  Critical  Operational  Issues  and  Criteria  (COIC) 
for  TRADOC  proponent  ACAT  I  and  II,  OSD  level  Theater/Tactical 
MAISRC  IMA,  and  OSD/DA  T&E  Oversight  systems.  Forwards  those 
applicable  to  Milestone  (MS)  I  TEMP  to  the  PM  for  inclusion  in 
the  TEMP.  Forwards  those  for  MS  II  and  system  modification  TEMP 
updates  to  HQDA  for  ADCSOPS-FD  approval. 

f.  Signs  TEMP  cover  page  as  user  representative  reviewer  for 
TRADOC  proponent  ACAT  I  and  II,  OSD  Theater/Tactical  IMA,  and 
OSD/DA  T&E  Oversight  systems. 

g.  Provides  member  to  the  ATEC. 

h.  Approves  Training  TSP  for  TRADOC  proponent  ACAT  I  through 
IV  materiel  and  Classes  1  through  5  Theater /Tactical/ IMA  systems. 

i.  Performs  staff  supervision  of  TRADOC  T&E  operations. 

j.  Approves  requests  for  waiver  of  operational  tests  and 
forwards  through  OPTEC  to  the  decision  authority  for  approval. 

3-30.  TRADOC  Proponent  Commands,  Centers,  Schools  and  Battle 
Labs 

a.  For  systems  in  acquisition  (particularly  ACAT  I  and  II 
materiel  systems,  Theater/Tactical  MAISRC  systems  and  OSD  T&E 
oversight  systems,  define  the  Force  Development  Evaluation  (FDEV) 
strategy  (including  requirements  for  CEP  and  FDTE)  to  support 
TRADOC  products  development,  maturation,  and  verification  prior 
initial  operational  test. 

b.  Serves  as  operational  evaluator  for  proponent  TRADOC 
product  FDEV  projects  (includes  definition  of  applicable 
operational  issues  and  criteria,  writing  chapters  1  and  2  of  TEP 
when  required  (e.g. ,  mutiple-source  evaluations  always  and  CEP 
almost  never  (chapter  one  provided  in  resume  sheet)  conducting 
the  evaluation,  and  reporting  the  evaluation) . 

c.  Develops  Resume  Sheet  (RS)  for  identified  CEP  requirements 
in  coordination  with  the  TECO  or  TEXCOM  (when  necessary)  and 


3-18 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992 


(DRAFT) 


submits  to  HQ  TRADOC  for  project  approval  and  prioritization. 


d.  Requests  designation  of  tester  by  OPTEC  for  those  FDTE  not 
addressed  in  a  TEMP  (must  be  accompanied  by  a  draft  TEP  chapter 
!)• 


e.  Develops  COIC  for  all  assigned  proponent  materiel  and  IMA 
systems  in  acquisition  and,  when  applicable,  modifications  to 
these  systems.  Approves  those  for  ACAT  III  and  IV  materiel 
systems  as  well  as  non— MAISRC  IMA  system  not  on  the  OSD  T&E 
oversight  list.  Forwards  to  HQ  TRADOC  for  approval  COIC  for  ACAT 
I  and  II  materiel  systems.  Theater /Tactical  MAISRC  systems,  and 
other  OSD/DA  T&E  oversight  systems. 

f.  Prepares  and  delivers  to  the  operational  tester  Doctrine 
and  Organization  Test  Support  Packages  (D&O  TSP) ,  Threat  TSP,  and 
Training  TSP.  Approves  D&O  TSP.  Forwards  Threat  TSP  for  ACAT  I, 
II,  and  OSD  T&E  oversight  systems  to  THRU  CAC  to  DA  (DCSINT)  for 
approval.  Approves  Threat  TSP  for  ACAT  III  and  IV  systems  not  on 
the  OSD  T&E  oversight  list.  Forwards  Training  TSP  to  HQ  TRADOC 
for  approval.  Provides  the  appropriate  Operational  Test 
Readiness  Statement  (OTRS  with  each  TSP. 

g.  Prepares  and  presents  COIC— ORD  Crosswalk  briefing  to 
ADCSOPS-FD  for  COIC  approval. 

h.  Serves  as  the  principal  Combat  Developer,  Training 
Developer,  and  Trainer  member  to  the  TIWG,  RAM  Scoring 
Conference,  RAM  Assessment  Conference,  Data  Authentication  Group 
(DAG) ,  and  Operational  Test  Readiness  Review  (OTRR) . 

i.  Conducts  (or  over  sees  conduct  of)  player  training 
(individual,  collective,  and  unit  training  for  operational 
testing.  Certifies  players  trained  and  ready  for  operational 
test. 


k.  Submit  requests  for  waiver  of  approved  TEMP  identified 
operational  tests  to  HQ  TRADOC  for  approval  and  forwarding 
through  OPTEC  to  the  decision  authority  for  approval. 

l.  Utilize  CEPs  to  identify  high  pay-off  technology  and  force 
development  initiative  early  and  concentrate  resources  on 
concepts  with  the  greatest  potential. 

3-31.  TRADOC  Analysis  Command  (TRAC) 

TRAC  (Fort  Leavenworth,  KS)  coordinates  TRADOC  centers  and 
activities  concerned  with  research  and  analysis.  The  analyses, 
modeling,  and  research  performed  by  TRAC  and  its  subordinate 
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activities  may  indicate  issues  or  estimate  outcomes  useful  in 
operational  evaluation. 

Section  V 

U.S.  Army  Materiel  Command  (USAMC) 

3-32.  USAMC  General 

AMC  is  the  Army's  principal  MATDEV.  Its  functions  include 
research,  development,  configuration  management,  DT&E,  production 
T&E,  establishment  of  a  common-use  T&E  database,  ILS  planning  and 
execution,  RAM,  acquisition  and  procurement,  production,  product 
assurance,  safety  and  health  considerations,  human  factors 
engineering  (HFE) ,  security,  and  new  equipment  training  (NET)  of 
materiel  systems. 

a.  Specific  AMC  functions  in  support  of  DT&E. 

(1)  Support  the  development  and  promulgation  of  DT&E  policy. 

(2)  Plan,  coordinate,  and  provide  functional  support  to  PEOs 
and  PMs,  and  execute  the  responsibilities  of  a  PM  for  assigned 
systems . 

(3)  Develop  system  threat  assessment  reports  after  MS  I  and 
Threat  System  Support  Packages  for  developmental  testing. 

(4)  Plan  for,  develop,  and  provide  specific  test  items  and 
support  materiel  to  conduct  required  required  testing. 

(5)  Include  the  impact  of  DT&E  in  the  system's  environmental 
assessment  (EA)  and  environmental  impact  statement  (EIS)  (AR 
200-2  and  AR  70-1) .  Ensure  that  the  developmental  tester  is 
informed  of  any  adverse  environmental  impacts  associated  with 
equipment  operation. 

(6)  Include  funds  for  DT&E  in  plans,  programs,  and  budgets. 

(7)  Provide  test  ranges  and  facilities  as  approved  in  the 
FYTP  to  support  testing. 

(8)  Provide  a  system  that  is  ready  for  DT&E.  Develop  exit 
criteria  for  each  milestone  and  T&E  event. 

(9)  Provide  a  member  to  the  ATEC  and  the  TSARC. 

b.  Specific  AMC  functions  in  support  of  OT&E. 

(1)  Monitor  and  support  OT&E  and  CE. 
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(2)  Develop  and  provide  for  the  OT&E  agency  a  NET  support 
package  (NETSP) ,  system  support  package  (SSP) ,  OTRSs,  and  system 
safety  releases  (SRs) ,  as  required. 

(3)  Include  the  impact  of  OT&E  in  the  system's  environmental 
assessment  (EA)  and  environmental  impact  statement  (EIS)  (AR 
200-2  and  AR  70-1).  Ensure  that  the  operational  tester  is 
informed  of  any  adverse  environmental  impacts  associated  with 
operational  use. 

(5)  Include  funds  for  OT&E  in  plans,  programs,  and  budgets. 

(6)  Provide  DT  data  to  the  OT&E  agency  and  arrange  for  OT&E 
agency  participation  in  DT  to  support  OT  and  CE  objectives. 

(7)  Provide  test  ranges  and  facilities  as  approved  in  the 
FYTP  to  support  OT&E  and  CE. 

(8)  Provide  a  system  that  is  ready  for  OT&E.  Develop  exit 
criteria  for  each  milestone  and  special  operational  T&E  event. 

3-33.  AMC  Commodity  Commands 

Six  of  the  ten  MSCs  of  AMC  are  the  commodity  commands,  which  are 
the  life  cycle  managers  for  the  materiel  in  their  commodity 
areas.  Life  cycle  management  includes  research  and  development, 
procurement,  logistic  support,  rebuild  direction,  and  disposal. 
Several  commands  operate  arsenals  and  plants  with  manufacturing 
and  production  capabilities.  The  commodity  commands  are: 

a.  Armament  Munitions  and  Chemical  Command  (AMCCOM) . 

b.  Aviation  and  Troop  Command  (ATCOM) . 

c.  Communications-Electronics  Command  (CECOM) . 

d.  Missile  Command  (MI COM) . 

e.  Tank-Automotive  Command  (TACOM) . 

f.  Simulations,  Training,  and  Instrumentation  Command 
(STRICOM) . 

3-34.  U.S.  Army  Test  and  Evaluation  Command  (TECOM) 

a.  Developmental  testing  functions.  As  the  Army's  principal 
developmental  tester  for  weapons  and  equipment,  TECOM  performs 
the  following  DT  functions: 
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(1)  Planning,  executing,  and  reporting  the  results  of  DT  of 
Army  materiel  for  each  acquisition  program.  DT  includes  early 
research  tests,  engineering  development  tests,  TFTs,  PPQTs,  PQTs, 
joint  development  tests  (JDTs) ,  and  contractor  and  foreign  tests 
(non-Government  testing  agreed  to  and  integrated  into  the  TEMP  to 
provide  data  for  technical  evaluation  purposes) .  TECOM  also 
conducts  post-production  testing,  e.g.,  stockpile  reliability  and 
reconditioning  tests  (AR  73-1) . 

(2)  Providing  test  facilities  for  the  conduct  of  DT. 

(3)  Serving  as  a  member  of  the  TIWG  to  provide  technical 
expertise  in  support  of  the  T&E  life  cycle. 

(4)  Maintaining  the  Army's  Major  Range  and  Test  Facility  Base 
(except  for  Kwajalein  Missile  Range) . 

(5)  Providing  Safety  Releases  to  testers  prior  to  any  tesing 
using  soldiers  as  test  players  (except  for  systems  developed  by 
ISC,  HSC,  SDC,  and  MRDC)  (AR  385-16). 

(6)  Providing  safety  confirmations  in  support  of  the  decision 
review  (AR  385-16) . 

(7)  Researching,  developing,  and  acquiring  test 
instrumentation;  developing  and  acquiring  advanced  test 
technologies  such  as  artificial  intelligence,  robotics,  directed 
energy,  and  space  test  procedures;  and  developing  new  and 
improved  test  methodology  to  increase  the  efficiency,  validity, 
and  reliability  of  DT. 

b.  Independent  development  assessor  functions.  In  addition, 
TECOM  performs  the  following  functions  as  AMC's  developmental 
assessor  for  ACAT  III  and  IV  programs: 

(1)  Planning,  performing,  and  reporting  CE  of  Army  materiel 
for  each  acquisition  program.  Developmental  evaluation  addresses 
the  system's  technical  parameters  and  the  acquisition  and 
fielding  of  an  effective,  supportable,  and  safe  system.  It  does 
this  by  assisting  in  engineering  design  and  development; 
verifying  the  adequacy  of  the  technical  data  package  and  the 
attainment  of  technical  performance  specifications,  objectives, 
producibility ,  and  supportability ;  and  determining  safety,  health 
hazards,  human  factors,  and  MANPRINT  aspects.  Developmental 
evaluations  encompass  the  use  of  models,  simulations,  and 
testbeds,  as  well  as  prototypes  or  full-scale  development  models. 

(2)  Assisting  the  developers  by  providing  information  about 
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technical  performance;  qualification  of  components; 
compatibility,  interoperability,  vulnerability,  transportability, 
and  survivability;  RAM;  MANPRINT;  system  safety;  ILS;  correction 
of  deficiencies;  accuracy  of  environmental  documentation;  and 
refinement  of  requirements. 

(3)  Confirming  readiness  for  OT  by  ensuring  that  the  system 
is  stressed  to  at  least  the  levels  expected  in  the  OT 
environment . 

(4)  Determining  that  the  system  is  technically  operable  in 
the  required  climatic  and  realistic  battlefield  environments, 
including  natural,  induced,  and  countermeasure  environments. 

(5)  Providing  a  representative  to  TIWGs  and  a  voting  member 
to  RAM  S/A  conferences. 

c.  Other  major  testing  functions. 

They  design,  conduct,  and  assess  developmental  tests.  TECOM 
testing  may  identify  issues  for  consideration  in  operational 
testing.  TECOM  data  and  reports  make  significant  contributions  to 
CE.  TECOM's  system  safety  activities  are  of  vital  importance  to 
the  T&E  community,  because  no  testing  involving  troops  may  begin 
until  the  system  is  certified  safe  for  use  under  the  conditions 
or  limitations  specified  in  the  SR  issued  by  TECOM.  TECOM  also 
provides  a  bridge  to  the  early  activities  of  the  MAP  through  the 
advanced  test  technology  interfaces.  TECOM  has  the  largest  and 
most  complete  assemblage  of  testing  technology  assets  in  the 
Army,  including  procedures,  instrumentation  and  equipment,  and 
skilled  scientists.  HQ  TECOM  is  located  at  Aberdeen  Proving 
Ground,  MD.  In  addition,  TECOM  has  nine  major  DT  activities 
around  the  country.  For  additional  information  on  the 
capabilities  of  these  test  activities,  see  Part  Four,  Section 
XXI. 

(1)  Aviation  Technical  Test  Center  (ATTC) ,  AL.  ATTC  does 
aviation  testing  of  fixed  wing  and  rotary  wing  aircraft,  aircraft 
components,  and  related  ground  support  equipment. 

(2)  Cold  Regions  Test  Center  (CRTC) .  CRTC  is  the  cold  region 
environmental  test  center  for  the  Army,  testing  all  commodities 
of  Army  systems. 

(3)  Combat  Systems  Test  Activity  (CSTA) ,  Aberdeen  Proving 
Ground,  MD.  CSTA  is  a  multipurpose  proving  ground  where  most 
major  commodities  are  tested  including  vehicles,  munitions, 
weapons,  general  equipment,  and  CIE.  Test  facilities  include 
firing  ranges,  complex  vehicle  test  courses,  a  radar  tracking 
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site  facility,  and  many  special  laboratories. 

(4)  Dugway  Proving  Ground  (DPG) ,  UT.  DPG  has  the  full  range 
of  capabilities  for  testing  chemical  agents  and  chemical  and 
biological  defensive  systems  and  equipment.  Facilities  also 
include  a  remote  test  site  in  Panama  to  test  the  full  range  of 
Army  systems,  clothing  and  individiual  equipment  for  effects  of 
operationa  and  long-term  exposure  in  natural  tropical 
environments. 

(5)  Electronics  Proving  Ground  (EPG) ,  AZ.  EPG  is  a  highly 
specialized  proving  ground  for  electronics-oriented  testing. 
Research  is  also  performed  in  image  technology,  image 
interpretation,  radar  applications,  antennae,  and  other  fields 
related  to  communications  and  electronics. 

(6)  Jefferson  Proving  Ground  (JPG),  IN.  JPG  is  the  Army's 
volume— product ion— level  ammunition  acceptance  test  facility 
capable  of  supporting  wartime  production.  It  has  a  5-mile  firing 
front  with  125  firing  sites  and  a  20,000-meter  range. 

(7)  White  Sands  Missile  Range  (WSMR) ,  NM.  WSMR  performs  DT 
of  Army  missile  systems,  air  defense  fire  distribution  systems, 
space  systems,  surface-to-surface  missile  systems,  and  other 
weapon  systems. 

(8)  Yuma  Proving  Ground  (YPG) ,  AZ.  YPG  is  a  multipurpose 
proving  ground  specializing  in  testing  of  long-range  artillery, 
aircraft  armament  systems,  air  delivery  systems,  and  in— desert 
environmental  testing.  YPG  operates  an  air-to-ground  and 
ground-to-ground  aircraft  armament  test  range  with  electronic  and 
optical  real-time  instrumentation. 

(9)  Redstone  Technical  Test  Center  (RTTC) ,  AL.  RTTC  tests 
small  rockets  and  guided  missiles.  Facilities  are  also  available 
for  testing  of  electro-optical  devices  under  simulated  battle¬ 
field  countermeasures  and  obscurants. 

3-35.  U.S.  Army  Materiel  System  Analysis  Activity  (AMSAA) 

AMSAA  is  a  systems  analysis  and  modeling  activity  reporting  to 
AMC. 


a.  As  the  independent  developmental  evaluator  for  Army 
materiel  systems,  AMSAA 's  responsibilities  include  participating 
in  the  CE  process  by  performing  the  following: 

(1)  Reviews  T&E  Master  plans  for  adequacy,  and  prepares  the 
developmental  T&E  portion  of  the  T&E  Master  Plan  (TEMP)  in 
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conjunction  with  the  developmental  tester. 

(2)  Prepares  Independent  Evaluation  Plans  (lEP)  for  DT. 

(3)  Prepares  overall  DT  Test  Design  Plans  (TDP) . 

(4)  Participates  in  the  development  of  Failure  Definition  and 
Scoring  Criteria  (FD/SC) . 

(5)  Participates  in  TIWGs. 

(6)  Voting  member  in  Scoring  Conferences  for  DT  and  OT. 

(7)  Participates  in  RAM  Assessment  Conferences. 

(8)  Analyzes  and  evaluates  DT  test  data. 

(9)  Conducts  risk  analyses. 

(10)  Performs  independent  DT  evaluations,  and  publishes 
Independent  Evaluation  Reports  (lERs) . 

(11)  Participates  in  OT&E  DAGs. 

(12)  Presents  technical  status/risk  assessments  at  OTRRs. 

(13)  Develops  exit  criteria  for  a  weapon  system  to  be 
certified  as  ready  for  OT&E. 

(14)  Develops  the  system  reliability  growth  curve  for  the 
system  LCC. 

b.  As  the  Army's  independent  logistician,  AMSAA: 

(1)  Performs  the  ILS  program  surveillance  for  Army  materiel 
systems . 

(2)  Performs  ILS  assessments. 

(3)  Evaluates  all  materiel  acquisition  programs  and  deployed 
systems,  except  for  medical  items, 

(4)  Monitors  tests  on  an  exception  basis. 

3-36.  Army  Research  Laboratory  (ARL) 

The  ARL  is  a  new  organization  within  AMC  which  consists  of  the 
seven  laboratories  which  made  up  the  former  Army  LABCOM. ^  It 
combines  the  laboratories  with  additional  selected  technical  base 
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activities  from  other  organizations  within  the  Army.  ARL  will 
eventually  consolidate  all  its  elements  into  two  locations  in  MD 
with  a  site  for  large  scale  experiments  and  outdoor  assessments 
in  NM.  ARL  is  a  world-class  laboratory  established  to  conduct 
basic  and  applied  research,  exploratory  development  and  analyses 
with  the  critical  mass,  synergy,  and  flexibility  to  satisfy  the 
Army's  future  technological  needs  in  sensors,  signatures,  signal 
and  information  processing;  electronics  and  power  sources; 
battlefield  environments;  vehicle  propulsion;  materials;  vehicle 
structures;  weapons  technology;  human  research  and  engineering; 
advanced  computing  and  software;  and  survivability/ lethality  and 
MANPRINT  analyses.  With  emphasis  on  teamwork  and  partnership 
with  industry  and  academia,  ARL  seeks  out  all  available  resources 
in  the  Army's  pursuit  to  give  our  soldiers  a  technological 
advantage . 

Section  VI 

Other  HQDA  Activities 

3-37.  Information  Systems  Command  (ISC) 

ISC  (Fort  Huachuca,  AZ)  participates  in  the  acquisition  process 
and,  as  designated  by  HQDA,  conducts  DT  and  OT  for  systems 
applicable  to  ISC's  mission.  Primary  focus  is  on  communications, 
electronic,  and  automatic  data  processing  (ADP)  equipment. 
Specific  responsibilities  are  in  AR  10-87  and  AR  73-1.  As  a 
CBTDEV,  ISC  establishes  materiel  development  objectives  and 
requirements.  As  a  tester,  ISC  conducts  DT  and  OT  and 
evaluations  of  communications  equipment  developed  for  use  in  the 
following: 

a*  Defense  Communications  System  (DCS),  Army. 

b.  Installation  level  communications. 

c.  Air  traffic  control  systems. 

d.  Other  communications  as  specifically  designated  by  HQDA. 

3-38.  Intelligence  and  Security  Command  (INSCOM) 

INSCOM  (Fort  Belvoir,  VA)  conducts  DT&E  and  OT&E  for  assigned 
classified  or  secured  systems  (See  AR  10-87  and  AR  73-1) . 

Assesses  the  functionality  and  interoperability  of  INSCOM  mission 
systems  (as  opposed  to  communications  systems  that  ISC  is 
responsible  for) .  INSCOM  is  the  CBTDEV  for  strategic  SIGINT 
systems  and  represents  DCSINT  on  study  advisory  groups  (SAGs) , 
special  task  forces  (STFs) ,  and  special  study  groups  (SSGs) . 
Specifically,  INSCOM: 
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a.  Establishes  materiel  development  objectives  and 
requirements . 

b.  Conducts  DT  and  OT  of  SIGINT  equipment  for  use  in  the 
National  SIGINT  System  (Army)  as  specifically  designated  by  HQDA 

c.  Is  responsible  for  the  overall  design  of  SIGINT  systems 
that  have  sole  application  to  the  SIGINT  System. 

d.  Coordinates  with  the  Commanding  General,  AMC,  on  matters 
related  to  acquiring  SIGINT  systems  for  which  INSCOM  has  been 
designated  the  MATDEV. 

3-39.  Health  Services  Command  (HSC) 

HSC  (Fort  Sam  Houston,  TX)  performs  OT&E  of  medical  materiel  (AR 
40-60) .  Specific  responsibilities  are  in  AR  10-87  and  AR  73-1. 
Through  the  HSC  subordinate  element,  the  Academy  of  Health 
Sciences,  HSC  is  the  medical  materiel  CBTDEV,  trainer,  user 
tester,  and  user  representative.  Specifically,  HSC: 

a.  Conducts  OT&E  of  medical  equipment  as  directed  by  ODCSOPS 

b.  Conducts  medical  materiel  combat  development  activities. 

c.  Reviews  and  evaluates  CBTDEV 's  documents  to  determine 
potential  health  hazards  related  to  nonmedical  materiel  under 
development. 

d.  Provides  technical  assistance  to  the  MATDEV  by  evaluating 
health  hazards  related  to  operating  materiel  systems. 

e.  Prepares  health  hazard  assessment  reports  to  TSG. 

3-40.  U.S.  Army  Forces  Command  (FORSCOM) 

FORSCOM  is  the  primary  source  of  user  troops  for  OT,  and  also 
provides  troops  for  DT,  when  required.  Because  FORSCOM  is  the 
ultimate  user  of  new  materiel,  its  participation  in  T&E  is 
essential  throughout  the  MAP.  In  the  program  initiation  phase, 
FORSCOM  ensures  that  its  interests  as  the  ultimate  user  of  the 
equipment  are  considered  during  MAAs.  Throughout  the  MAP, 
FORSCOM  refines  requirements  for  user  troops.  During  the 
production  and  deployment  phase,  FORSCOM  provides  user  comments, 
usage  data,  and  requests  for  product  improvement. 

3-41.  Concepts  Analysis  Agency  (CAA) 

CAA  is  a  field  operating  agency  that  reports  to  the  Director  of 
the  Army  Staff,  Office  of  the  Chief  of  Staff,  Army  (OCSA) .  Its 
mission  is  to  conduct  analyses  of  Army  systems  in  the  context  of 
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joint  or  combined  forces.  These  analyses  establish  the  context 
for  lower— level  systems  and  may  establish  Issues  and  Criteria 
useful  in  OT&E.  CAA  provides  data  from  force  design  or 
requirement  studies  and  formulates  performance  data  requirements 
for  FDTE.  CAA  contributes  to  the  CE  program  through  its 
modeling,  simulation,  and  studies  efforts.  CAA's  functions  and 
responsibilities  are  detailed  in  AR  10-38. 

3-42.  Logistics  Evaluation  Agency  (LEA) 

LEA  is  a  field  operating  agency  of  ODCSLOG.  The  Commander, 

USALEA  will  assist  OPTEC  with  evaluation  or  assessment  of  IMA 
logistics  systems  under  MAISRC  control  which  are  processed  on 
selected  tactical  computer  systems. 

Section  VII 
Program  Sponsors 

3-43.  Program  Executive  Officer  (PEO)  (See  AR  70-1,  AR  25-3.) 

The  PEOs  review  and  provide  their  assessment  of  any  changes 
'reported  in  assigned  individual  programs,  the  significance  of 
problems  reported  by  the  Program  Manager,  the  Program  Manager's 
proposed  action  plans,  and  the  level  of  risk  associated  with  such 
plans.  The  PEO  can  be  the  program  sponsor  as  well  as  the  MATDEV. 
The  PEO  is  responsible  for  the  following: 

a.  Administer  a  defined  number  of  assigned  major  and  nonmajor 
programs,  as  approved  by  the  AAE,  ensuring  that  all  Army  agencies 
are  responsive  to  the  needs  of  the  PM  in  achieving  programmatic 
goals. 


b.  Charter,  supervise  and  evaluate  assigned  program,  project, 
and  product  managers  and  provide  the  planning  guidance, 
direction,  control,  oversight,  and  support  necessary  to  field 
systems  within  cost,  schedule,  and  performance  baselines. 

c.  Provide  technical,  operational,  and  functional  integration 
across  assigned  programs.  Ensure  that  functional  (matrixes) 
support  to  subordinate  PMs  is  planned  and  coordinated  with  the 
supporting  organizations. 

3-44.  Program  Managers 

a.  Program  Managers.  The  actual  manager  of  a  specific 
acquisition  or  development  program  at  its  inception,  normally 
referred  to  as  PM,  Product  Manager,  or  Project  Manager  ensures 
the  planning,  coordination,  and  support  of  the  T&E  activities  of 
the  specific  program  by: 
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(1)  Providing  assessments  of  program  status  and  risk  in  all 
briefings  and  presentations  to  higher  authorities,  actively 
manage  contract  performance,  and  provide  assessment  of  contractor 
performance. 

(2)  Preparing,  coordinating,  distributing,  and  maintaining 
the  TEMP. 

(3)  Establishing  and  chairing  a  Test  Integration  Working 
Group  (TIWG)  to  effect  T&E  coordination  and  solve  routine 
problems.  Substantive  issues  which  cannot  be  resolved  by  the 
TIWG  will  be  surfaced  through  the  chain  of  command  for 
resolution. 

(4)  Providing  the  testers  and  evaluators  the  opportunity  to 
participate  in  the  preparation  of  the  testing  portion  of  the 
Request  for  Proposal  (RFP)  to  assure  that  T&E  requirements  are 
accurately  reflected  in  contractual  documents.  Changes  occurring 
during  the  contract  negotiations  which  affect  testing  will  be 
communicated  to  the  TIWG.  The  TEMP  will  be  updated  to  reflect 
those  changes. 

(5)  Conducting  Development  Test  Readiness  Reviews  (DTRR) ,  and 
preparing  a  Development  Test  Readiness  Statement  (DTRS)  verifying 
that  the  system  is  ready  for  the  Preproduction  Qualification  Test 
(PPQT)  for  materiel  systems,  or  the  Software  Qualification  Test 
(SQT)  for  ISs. 

(6)  Providing  formal  certification  (via  an  Operational  Test 
Readiness  Statement  (OTRS)  that  the  system  is  ready  for 
operational  test.  Upon  request  from  the  user,  providing  formal 
certification  that  the  system,  to  include  brassboards  in  the 
development  stage,  is  ready  for  use  in  the  Force  Development  Test 
and  Experimentation  (FDTE) . 

(7)  Assuring  that  DT&E  of  all  systems  are  conducted  in 
representative  environments  to  include  testing  in  natural 
environments . 

(8)  Establishing,  documenting,  and  providing  the  system 
support  package  (SSP)  and  new  equipment  training  support  package. 

(9)  Considering  use  of  U.S.  Army  Materiel  Command  (USAAMC) , 
Test  and  Evaluation  Command  (TECOM)  test  facilities  before 
establishing  in-house  capabilities  or  contracting  for  testing 
services.  It  is  recognized  that  there  will  be  instances  when 
TECOM  test  facilities  cannot  accommodate  testing  or  scheduling 
requirements,  or  when  other  compelling  reasons  exist  that  would 
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make  use  of  TECOM  facilities  impractical.  om 
should  be  documented  in  the  TEMP  and  be  coordinated  with  TECOM, 
the  TIWG  and  TEMP  approval  authorities. 


(10)  Determining  (in  coordination  with  the  TIWG)  the  degree 
to  which  LFT&E  will  be  conducted  for  designated  systems. 


(11)  Participating  in  assigned  TAIGs. 

b.  Programs.  PM  managed  programs  are  categorized  as 
executive  and  non-executive • 


(1)  An  executive  program  is  managed  by  a  PM  who  is 
subordinate  to  a  PEO  or  a  PM  reporting  directly  to  the  AAE.  All 
major  systems  and  Army  high  interest  programs  will  be  executive. 


(2)  A  non-executive  program  is  managed  by  a  PM  subordinate  to 
any  other  materiel  developer  (e.g.,  U.S.  Army  Materiel  Coimand, 

US  Army  Corps  of  Engineers,  U.S.  Army  Medical  Research  and 
Development  Command,  U.S.  Army  Information  Systems  Command.) 

c.  The  requirement  for  all  PMs  will  be  established  by  the 
AAE.  The  executive  PMs  will  be  chartered  by  and  report  directly 

to  PEGS. 

3-45.  PM  Training  Devices  (PM  TRADE) 

The  PM  TRADE  (STRICOM)  is  the  advisor  and  independent  program 
assessor  for  all  training  device  developments.  Systems 
developers  will  coordinate  the  development  of  training  devices 
with  the  PM  TRADE  before  proceeding.  (See  AR  350-38). 

3-46.  PM  instrumentation.  Targets  and  Threat  Simulators  (PM 

ITTS)  ...  ^  ...  ^ 

PM  ITTS  (STRICOM)  has  responsibility  for  oversight  or  aii 
operational  test  instrumentation  programs,  to  include  both 
and  sustaining  instrumentation.  PM  ITTS  also  coordinates  with 
OPTEC  to  address  operational  testing  requirements. 

Section  VIII 
Review  Fora 

3-47.  Defense  Acquisition  Board  (DAB) 

The  DAB  is  the  primary  forum  for  resolving  issues  and 
facilitating  Under  Secretary  of  Defense  for  Acquisition  decisions 
for  acquisition  category  I  programs.  In  support  of  the  Defense 
Acquisition  Board,  the  appropriate  Committee  of  the  Board  will 
conduct  a  pre— Defense  Acquisition  Board  review.  The  office  of 
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the  Secretary  of  Defense  Cost  Analysis  Improvement  Group  and  the 
Joint  Requirements  Oversight  Council  also  support  the  Defense 
Acquisition  Board  in  its  review  process.  The  DAB  is  chaired  by 
the  Under  Secretary  of  Defense  for  Acquisition,  and  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  serves  as  vice  chairman.  A 
more  detailed  discussion  on  the  DAB  process  and  procedures  is 
contained  in  DODI  5000.2,  Part  13. 

3-48.  Army  Systems  Acquisition  and  Review  Council  (ASARC) 

The  ASARC  is  the  Army's  senior-level  review  body  for  ACAT  I  and 
II  programs.  The  ASARC  will  be  convened  at  formal  milestones  to 
determine  a  program  or  system's  readiness  to  enter  the  next  phase 
in  the  materiel  acquisition  cycle,  and  make  recommendations  to 
the  AAE  on  those  programs  for  which  he/ she  is  the  MDA.  An  ASARC 
may  also  be  convened  at  any  time  to  review  the  program  status. 
ACAT  ID  programs  are  subsequently  reviewed  by  the  DAB.  It  is  co¬ 
chaired  by  AAE  and  VCSA.  ASARC  membership,  functions  and 
procedures  are  outlined  in  AR  70-1. 

3-49.  Major  Automated  Information  System  Review  Council  (MAISRC) 

a.  DoD  MAISRC.  The  DoD  MAISRC  is  chartered  by  the  Assistant 
Secretary  of  Defense  for  Command,  Control,  Communications,  and 
Intelligence  under  the  overall  guidance  of  DoD  Directive  5000.1, 
Defense  Acquisition,  and  operates  in  accordance  with  DoD 
Directive  7920.1,  Life  Cycle  Management  of  Automated  Information 
Systems  and,  DoD  Instruction  7920.2,  Automated  Information 
Systems  Life-Cycle  Management  Review  and  Milestone  Approval 
Procedures.  Automated  Information  Systems  that  meet  the 
thresholds  for  acquisition  category  I  programs  ^re  reviewed  by 
the  DAB. 

b.  Army  MAISRC.  Principal  HQDA  officials  will  participate  in 
the  Army  MAISRC  as  required.  The  MAISRC  recommends  actions  to  SA 
for  decision  or  subsequent  recommendations  to  SECDEF.  A  MAISRC 
provides  the  Army  position  for  DAB  consideration  for  major 
systems.  For  Class  I/II  systems  the  MAISRC  provides  the  SA  with 
its  recommendation  on  the  system.  MAISRC  functions  and 
procedures  are  outlined  in  DODI  7920.2  and  AR  25-3.  The 
appropriate  MAISRC  performs  a  formal  review  on  systems  subject  to 
AR  25-3.  The  purpose  of  the  review  is  for  management  to  obtain  a 
current  status  of  the  project  and  to  provide  additional  guidance 
and/or  give  milestone  approval  to  the  project.  All  class  II  Iss 
also  undergo  a  DOD  MAISRC  review  unless  the  responsibility  has 
been  delegated  to  the  Army.  The  criteria  that  determine  whether 
a  system  requires  formal  HQDA  review  are  in  Chapter  2,  AR  25-3. 
The  review  process  prescribes  a  set  of  steps 

with  specific  documentation  requirements.  It  provides  detailed 
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procedures  for  conducting  management  reviews,  describes  the 
actions  that  must  be  completed  prior  to  each  review,  and 
specifies  the  documentation  necessary  for  approval  of  an  ifa 
designated  for  MAI SRC  review. 

3-50.  In-Process  Review  (IPR)  . 

The  IPR  is  the  review  forum  for  all  ACAT  III  and  IV  programs  and 
is  chaired  by  the  Milestone  Decision  Authority  (MDA) .  General 
policies  for  reviews  of  IPR  programs  are  the  same  as  for  acai  i 
and  II  programs.  Reviews  are  conducted  at  formal  milestones 
at  other  times  deemed  necessary  by  the  MDA.  IPR  members  include 
the  MATDEV,  CBTDEV,  independent  operational  and 
evaluator,  logistician,  trainer  (if  different  from  the  CBTDEV), 
functional  support  organization  or  staff  '  ^%T>wvr 

determined  by  the  IPR  Chair.  For  selected  IPR  tests,  OPTEC 
monitors  OT&E  by  reviewing  and  commenting  on 
OPTEC  may  either  send  a  panel  member  to  present  the  results 
evaluations  directly  to  the  IPR  panel  or  forward  a  written 
Operational  Assessment/  Abbreviated  Operational  Assessment 
(OA/AOA) .  IPR  functions  are  covered  in  AR  70-1. 

3-51.  Materiel  Release  Review  Board  (MRRB) 

Materiel  release  policy  is  stated  in  AR  700-142,  Materiel 
Release,  Fielding,  and  Transfer.  The  testers  and  evaluators 
inform  the  MATDEV,  CBTDEV,  and  other  ILS  program  participants  of 
potential  materiel  release,  fielding,  or  transfer  problems  and 
recommend  solutions  to  the  problems.  The  independent  evaluators 
prepare  operational  assessments  or  test  and  evaluation  repor 
(TER),  that  present  a  position  relative  to  proposed  materiel 
releases.  The  TERs  address  the  ability  of  the  system  to  fulfill 
the  requirements  in  the  approved  requirements  documents  and 
specifications.  Materiel  release  prerequisites  must  be  met 
before  materiel  release.  To  ensure  that  the  objectives  of  the 
Lteriel  release  process  are  reached,  the  MATDEV  will  provide  the 
logistician,  CBTDEV,  and  other  participants  in  the  MRRB,  a  copy 
of  the  documentation  showing  that  the  materiel  release 
prerequisites  have  been  met. 

3-52.  Test  Schedule  and  Review  Committee  (TSARC) 

TSARC  is  the  formal  process  through  which  Outline  Test  Plans 
(OTPs)  are  approved  and  included  in  the  Five  Year  Test  Program 
(FYTP).  Established  by  CSA,  the  TSARC  provides  high-level 
centralized  resource  management,  by  maximizing  the  use  of  limited 
resources  and  minimizing  the  impact  on  unit  operational  readiness 
(AR  15-38).  Commanding  General,  OPTEC,  chairs  the  TS^C,  _ 
prepares,  coordinates,  and  presents  proposed  changes  to  the  FYTP, 
and  publishes  the  FYTP  after  ODCSOPS  approval.  OPTEC  also 
develops  policy  guidance  for  conduct  of  TSARC.  Commanding 
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General,  AMC,  and  all  operational  testers  other  than  OPTEC  submit 
coordinated  OTPs  to  the  TSARC  for  inclusion  in  the  FYTP. 

a.  Schedule.  The  TSARC  cycle  occurs  semiannually  from 
February  to  August  and  from  September  to  December  (see  table  2-3 
for  approximate  dates;  TSARC  establishes  specific  dates) .  OPTEC 
chairs  follow-on  DA  working  group  TSARCs  in  May  and  October.  DA 
General  Officer  TSARCs  are  held  in  April  and  December  to  consider 
issues  not  resolved  during  the  DA  working  group  TSARCs.  The 
General  Officer  TSARC  then  recommends  approval  of  OTPs  for 
inclusion  in  the  FYTP  to  ODCSOPS.  Procedures  for  approving  an 
OTP  between  scheduled  TSARCs  are  in  AR  15-38  and  the  TSARC 
Handbook . 

b.  Functions.  The  primary  functions  of  the  TSARC  are  to; 

(1)  Review  and  recommend  coordinated  OTP  for  inclusion  in  the 
FYTP. 

(2)  Review  and  recommend  UT/TT  priorities  and  resource 
support . 

(3)  Resolve  conflicts  between  test  requirements  and  other 
missions. 

(4)  Review  and  recommend  approval  of  the  FYTP. 

3-53.  Test  Integration  Working  Group  (TIWG) 

a.  Functions.  A  TIWG  is  chartered  for  acquisition  systems  in 
accordance  with  Draft  AR  73-'l,  to  coordinate  and  integrate  T&E 
planning  and  participation  by  all  parties.  The  TIWG  supports  CE 
by  sharing  data  and  doing  continuing  T&E  documentation,  planning 
and  integration  of  T&E  in  the  development  and  acquisition 
process.  The  TIWG  does  not  alter  the  responsibilities  or 
authority  of  any  command  or  headquarters.  In  the  event  of 
disagreements,  issues  are  resolved  through  normal  command 
channels.  A  more  detailed  list  of  TIWG  functions  are  in  AR  73- 
XX ,  Chapter  6 . 

b.  Establishment.  The  TIWG  is  established  by  the  MATDEV  upon 
program  initiation  (e.g.,  receipt  of  the  acquisition  decision 
memorandum  (ADM)  or  approval  of  the  MNS) .  It  supports 
preparation  of  the  TEMP  and  addresses  the  T&E  aspects  of  the  AS, 
RFP  or  contract,  and  other  program  management  documents  (AR 
70-1) .  TIWG  members  are  members  of  the  Acquisition  Team  and 
remain  a  principal  active  working  group  throughout  the  MAP.  TIWG 
members  also  coordinate  on  waivers  of  required  testing. 
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c.  Membership.  Membership  in  the  TIWG  includes  the;  MATDEV 
(chair),  CBTDEV,  Technical  tester.  Technical  evaluator, 

Operacional  tester.  Operational  evaluator.  Logistician,  Trainer, 

(1)  The  Army  Command  Control  System  (ACCS)  systems  engineer 
or  subsystem  engineer  for  any  ACCS  component  system  or  eguipment 
that  has  one  or  more  interfaces. 

(2)  The  PM  for  smoke  or  obscurants  for  all  systems  that  rely 
on  electromagnetic  propagation  or  are  susceptible  to  aerosol 
countermeasures . 

d.  The  associate  members  are; 

(1)  The  Integrated  Logistics  Support  Management  Team  (ILSMT) . 

(2)  The  contractor. 

(3)  Monitoring  commands  or  activities  (e.g.,  TSG's 
representative  for  health  aspects  associated  with  testing) . 

(4)  Subcommands  or  activities  of  principal  attendees. 

e.  Subgroups.  At  least  the  following  subgroups  are  formed  to 
support  the  TIWG; 

(1)  The  RAM  subgroup  is  chaired  by  the  MATDEV  and  includes  at 
least  the  CBTDEV  and  the  independent  technical  and  operational 
evaluators.  The  purpose  of  this  subgroup  is  to  address  all 
RAM-related  matters. 

(2)  The  supportability  T&E  subgroup  is  chaired  by  the  ILS 
manager  of  the  MATDEV.  The  purpose  of  this  subgroup  is  to 
provide  coordination  with  the  ILSMT.  Topics  to  be  coordinated 
include  all  supportability  test  issues,  test  requirements, 
transportability,  and  logistic  demonstration  requirements 
contained  in  the  TEMP. 

f.  Other  groups  interfacing  with  the  TIWG.  The  TIWG  may 
interface  with  the  following  working  groups  during  the  life  cycle 
of  the  system; 

(1)  RAM  rationale  report  working  group  (RRRWG) . 

(2)  Computer  resources  working  group  (CRWG) . 

(3)  MANPRINT  joint  working  group  (MJWG) . 
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(4)  Technical  test  readiness  review  (TTRR)  working  group. 

(5)  OTRR  working  group. 

(6)  Data  authentication  group  (DAG) . 

(7)  Joint  Tactical  Communications,  Command,  and  Control 
Agency . 


(8)  Threat  Coordinating  Group  (TCG) . 

(9)  Threat  Accreditation  Working  Group  (TAWG) . 

3-54.  Test  and  Analysis  Integration  Group  (TAIG) 

A  TAIG  is  required  after  MS  0  for  all  ACAT  ID,  IC,  II,  and  III/IV 
DoD  Oversight  programs  for  which  a  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  is  planned.  HQDA  DCSOPS  will 
establish  each  TAIG  and  its  membership.  Its  responsibilities 
include: 


a.  Plans  linkage  between  ORD  development,  COEA  study  plan  and 
Essential  Elements  of  Analysis  (EEA) ,  and  Critical  Operational 
Issues  and  Criteria  (COIC)  development. 

b.  Ensures  a  credible  crosswalk  between  the  TEMP  development 
the  ORD,  the  COEA,  the  COIC,  and  the  Additional  Operational 
Issues  and  Criteria  (AOIC)  in  the  TEP. 

c.  When  model-test-model  (MTM)  is  required,  the  TAIG  examines 
the  plans  for  that  effort,  and  recommends  as  necessary. 

d.  Through  the  materiel  developer  member  of  the  TAIG,  advises 
the  TWIG  to  incorporate  pertinent  analyses  into  TWIG  effbrts. 

3-55.  Operational  Test  Readiness  Review  (OTRR) 

OTRR's  are  conducted  by  the  operational  tester  prior  to  each 'test 
to  allow  the  tester  an  assessment  of  readiness  to  test  the  • 
system.  The  purpose  of  the  OTRR  is  to  determine  the  readiness  of 
the  system,  support  packages,  instrumentation,  test  planning,  to 
support  the  OT.  It  includes  identification  of  any  problems  which 
may  impact  the  start  or  adequate  execution  of  the  test.  The 
objective  of  the  review  is  to  determine  if  any  changes  are 
required  in  planning,  resources,  training,  equipment,  or  timing 
to  successfully  proceed  with  the  test.  OTRR's  are  chaired  by 
OPTEC  or  other  OT&E  activity.  Principal  attendees  include  the 
operational  tester,  operational  evaluator,  MATDEV,  CBTDEV, 

TNGDEV,  FORSCOM  (or  other  activity  providing  player  or  O^FOR 
troops) ,  logistician,  technical  tester,  technical  evaluators. 


3-35 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992 


(DRAFT) 


HQDA  staff  elements,  host  installation,  and  contractor.  Th 
primary  OTRR  is  conducted  prior  to  resource  deployment  to  test 
site  (T-60) .  Additional  reviews  are  conducted  as  required.  see 
Chapter  5  for  detailed  instructions  on  OTRR. 

3-56.  Special  Task  Force  (STF)  and  Special  Study  Groups  (SSG) 

An  STF  or  SSG  is  normally  formed  during  the  concept  exploration 
phase  and  is  convened  to  conduct  analyses,  ensure  that  all 
alternatives  are  included  in  the  analyses,  monitor  experimen  s, 
or  undertake  other  such  tasks  that  may  require  a  concentration  of 
special  expertise  for  a  designated  period  of  time.  The  SSG 
determines  what  developmental  instrumentation  technologies  will 
be  required  to  support  both  technical  and  operational  testing. 

The  STF  is  chartered  by  CSA  under  the  General  Staff  supervision 
of  ODCSOPS.  The  SSG  is  chartered  and  under  the  supervision  of 
the  Commanding  General,  TRADOC.  The  STF  or  SSG  reviews  the 
requirements  stated  in  the  MNS  or  Operational  Requirements 
Document  (ORD) .  The  functions  and  procedures  of  the  STF  and  SSG 
are  in  AR  71-9.  ODCSOPS  or  TRADOC  sends  the  final  report  of  the 
STF  or  SSG  to  the  proper  commands  and  agencies.  The  final  repor 
may  be  used  in  the  following: 

a.  Concept  formulation  package  (CFP) ,  including  MNS  (if 
developed)  and  COEA. 

b.  System  concept  paper  (SCP) . 

c.  Baseline  cost  estimate  (BCE) . 

d.  TEMP. 

e.  ILS  plan. 

f.  standardization  and  interoperability  plan. 

g.  Threat  support  plan. 

3-57,  Concept  Evaluation  Program  (CEP)  Schedule  and  Review 

Council  (CEPSARC)  • -I 

The  CEPSARC  is  a  TRADOC  operated  and  chaired  council  which  meets 

at  least  annually  to  review  and  prioritize  its  CEP 
new  submissions  and  previously  approved)  to  recommend  the  CEP 
program  to  the  Deputy  Chief  of  Staff  for  Combat  Developments 
(DCSC)  TRADOC  for  approval  to  execute.  Representatives  include 
TRADOC  HQ,  Commands,  Centers,  Schools,  and  Battle  Labs  (CEP 
proiects) ;  OPTEC  (ammunition  and  flying  hour  programs  impace) ; 
FORSCOM  (unit  impace  if  any)  and  TEXCOM  (test  organization 

impact) . 
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3-58.  Army  Test  and  Evaluation  Committee  (ATEC) 

a.  Providing  a  forum  by  means  of  which  all  elements  of 

the  Army  T&E  community,  acting  as  a  committee  of  the  whole,  may 
formulate  recommendations  to  the  Army  senior  leadership  regarding 
T&E  policy,  T&E  procedures,  organization,  and  resources. 

b.  Undertaking  studies  and  reviews  of  specific  T&E  matters 
such  as,  but  not  limited  to,  the  test  instrumentation  program, 
development  of  automated  test  data  retrieval  systems,  and  quality 
assurance  of  T&E  products  (test  plans,  designs,  etc.). 

c.  Coordinating  the  missions,  functions,  composition, 
responsibilities,  and  concept  of  operations  of  all  T&E  activities 
within  the  Army  and  interface  with  OSD. 

d.  The  ATEC  is  responsible  for  senior  level  focus  on,  and 
centralized  guidance  to,  the  management  and  coordination  for  all 
major  T&E  policy  and  resource  issues. 

e.  The  ATEC  is  responsible  for  the  following  functions: 

(1)  Develop  and  review  Army  T&E  policy  and  procedures. 

(2)  Review  and  approve  Army  T&E  requirements  and 
allocations  of  available  resources  across  functional  lines. 

(3)  Review,  forecast  and  prioritize  future  Army  T&E 
instrumentation  requirements. 

(4)  Review  and  coordinate  modernization  of  T&E  facilities. 

(5)  Ensure  proper  coordination  is  effected  between  the  T&E 
community  and  the  PEOs  such  that  requirements  for  T&E  resources, 
unique  to  a  specific  program,  are  resourced  by  the  PEG. 

3-59.  Big  Three  Meetings 

A  group  consisting  of  the  ADCSOPS-FD,  TRADOC  DCSCDD,  and  the 
OPTEC  Commander.  The  TEMA  Director,  the  OEC  Commander,  and  the 
TEXCOM  Commander  sit  as  associate  members.  The  group  meets 
periodically  to  discuss  problems  and  topics  of  interest  in  the 
area  of  user  testing  and  its  interface  with  combat  and  training 
developments . 

3-60.  OTA  Commanders  Conference 

Consists  of  the  OPTEC  Commander,  the  Navy  OPTEVOR  Commander,  the 
AFOTEC  Commander,  and  the  Marine  Corps  OTEA  Commander.  The  DOT&E 
is  often  represented  at  meeting  of  this  body.  The  group  meets 
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twice  a  year  to  discuss  topics  of  mutual  interest  relating  to 
OT&E. 

3-61.  RAM  Scoring  and  Assessment  Conferences 

Scoring  and  assessment  (S/A)  conferences  are  conducted  during  or 
immediately  following  testing  to  insure  effective  accounting  of 
RAM  test  data  for  use  in  evaluation.  The  operational  tester  and 
operational  evaluator  attend  all  S/A  conferences  which  address 
data  intended  to  support  effectiveness  and  suitability 
evaluations.  Furthermore,  the  operational  evaluator  chairs  the 
OT  scoring  and  assessment  conferences  and  makes  final  Army 
determinations  at  OT  conferences  when  no  majority  exists.  The 
MATDEV  chairs  DT  scoring  conferences,  and  the  independent 
development  evaluator  decides  final  Army  positions  on  scoring  RAM 
for  these  tests.  The  goals  of  the  RAM  Assessment  conference  are 
to  establish  a  common  data  base  and  to  estimate  the  extent  to 
which  the  system  has  met  all  RAM  requirements.  See  AR  702-3  and 
Chapter  11  for  details  of  RAM  Scoring  and  Assessment  Conferences. 

3-62.  Data  Authentication  Group  (DAG) 

The  DAG  is  a  team  of  independent  experts  with  a  broad  spectrum  of 
technical  disciplines  assembled  for  the  purpose  of  assessing  and 
monitoring  data  reduction,  quality  control,  and  the 
identification  and  analysis  of  anomalies  in  the  system,  instru¬ 
mentation,  and  test  data.  The  principal  goal  of  a  DAG  is  a 
validated  data  base  that  accurately  reflects  how  a  candidate 
system  performed  during  test.  See  chapter  9  for  details  of  DAG. 


DOTSE 


Figure  3-1.  TSE  comnnjnity  functional  Interactions. 


Extract  from  DoD  Directive  5000.1 


Each  military  Department  shall  establish  an  independent 
operational  test  and  evaluation  activity.  This  activity 
shall: 

—  Be  separate  and  independent  from  the  materiel- 
developing  and  materiel-procuring  agency  and  the  using  agency. 

--  Be  responsible  for  planning  and  conducting  operational 
tests,  reporting  results,  and  providing  evaluations  of  each 
tested  system's  operational  effectiveness  and  suitability. 

—  Report  directly  to  the  Secretary  of  the  Military^ 
Department  who  may  delegate  responsibility  for  supervising 
this  activity  to  the  Service  Chief  concerned. 

Figure  3-2 


I 
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Chapter  4  .... 

Test  and  Evaluation  in  Support  of  the  Materiel  Acquisition 

Process 

Section  I 
Introduction 


4-1.  General 

a.  T&E  is  an  essential  activity  in  support  of  the  Materiel 
Acquisition  Process  (MAP) .  It  plays  a  key  role  in  the  life  cycle 
of  Army  materiel  systems,  providing  information  that  assists  in 
selecting,  acquiring,  using,  and  disposing  of  Army  materiel.  T&E 
is  inherent  in  the  technology  base  activities  that  provide  new^ 
technologies  to  be  exploited;  it  is  used  to  support  the  selection 
of  best  solutions  to  satisfy  a  mission  area  deficiency;  it 
verifies  that  the  Army  is  designing,  developing,  producing,  and 
stockpiling  materiel  that  satisfies  the  users'  needs;  and  it 
assists  in  the  assuring  that  materiel  which  is  no  longer  usable 
can  be  disposed  of  safely. 

b.  Comprehensive  developmental  and  operational  testing  shall 
be  conducted  on  all  materiel  systems.  Early  detailed  T&E 
planning  is  critical  to  meaningful  evaluations  and  assessments, 
as  well  as  to  the  successful  development  of  the  system.  The  T&E 
strategy  shall  specify  the  impact  on  risk  of  the  technologies  and 
processes  selected  for  system  development  during  the  entire  life 
cycle  of  the  system. 

c.  Developmental  T&E  shall  be  planned  and  incorporated  into 
the  materiel  system's  development  process  to  verify  conformance 
to  contract  specifications  and  critical  technical  parameters  in 
order  to  meet  technical  objectives  and  requirements. 

Developmental  T&E  shall  encompass  the  system  hardware,  software, 
users  manuals,  training  material,  interfaces,  compatibility,  and 
interoperability  with  existing  or  planned  systems.  Operational 
T&E  shall  examine  system  effectiveness  and  suitability  under 
operationally  realistic  conditions  when  the  system  is  operated  by 
typical  users.  In  addition,  the  T&E  should  address  the  system's 
compatibility  and  interoperability  with  users  and  other  systems. 

d.  Continuous  evaluation  (CE)  as  discussed  in  Chapter  2  is  a 
major  ingredient  in  the  T&E  which  supports  the  MAP.  It  should 
begin  as  early  as  the  battlefield  functional  mission  area 
analysis  and  continue  through  the  materiel  system's 
postdeployment  activities. 
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e.  Detailed  information  concerning  T&E  of  materiel  systems  is 
contained  in  Parts  Four,  Five,  and  Six.  In  addition,  T&E  of 
materiel  software  is  contained  in  Part  Seven.  Roles  and 
procedures  for  all  elements  of  materiel  software  T&E,  including 
the  maintenance  of  software  after  deployment,  are  discussed. 

f.  The  phases  and  milestones  for  the  life-cycle  system 
management  model  (LCSMM)  for  materiel  systems  is  illustrated  in 
figure  4-1. 


(INSERT  FIGURE  4-1  HERE) 


Section  II 

T&E  Activities  During  Determination  of  Mission  Needs 


4-2.  Determination  of  Mission  Need  Activities 
All  acquisition  programs  are  based  on  the  identification  of 
mission  needs.  A  mission  need  may  be  to  establish  a  new 
operational  capability  or  to  improve  an  existing  capability.  If 
a  mission  need  cannot  be  satisfied  by  a  nonmateriel  solution 
(i.e.,  changes  in  doctrine,  operational  concepts,  tactics, 
training,  and  organization),  then  a  Mission  Need  Statement  (MNS) 
is  developed.  The  MNS  is  a  broad  statement  of  mission  need, 
expressed  in  terms  of  an  operational  capability  rather  than  a 
system-specific  solution.  This  phase  ends  at  Milestone  0  (MS  0) , 
which  formally  approves  the  MNS. 

a.  Key  Events.  In  this  phase,  the  Combat  Developer  (CBTDEV) 
determines  whether  a  mission  deficiency  or  an  opportunity  to 
improve  an  existing  system  is  important  enough  to  warrant  further 
analysis  and  development  of  a  system.  The  CBTDEV  ensures  that 
pj-Qper  planning  and  evaluation  are  successfully  carried  out.  Key 
activities  associated  with  the  determination  of  mission  needs 
process  are  depicted  in  figure  4-2. 

(INSERT  FIGURE  4-2  HERE) 

b.  T&E  Activities.  T&E  activities  during  this  phase  usually 
involve  the  evaluation  of  nonmateriel  solutions  to  satisfy  an 
identified  mission  need.  The  CBTDEV,  with  the  assistance  of  the 
independent  operational  evaluator,  may  execute  a  Concept 
Evaluation  Program  (CEP)  to  aid  in  this  evaluation.  The  CEP  can 
provide  the  CBTDEV  with  a  quick  reaction  and  simplified  process 
to  examine  and  resolve  combat  development,  doctrinal,  and 
training  issues.  Within  a  CEP,  tests  may  be  executed  to  provide 
experimental  databases  for  an  MNS  and  subsequent  requirements 
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documents.  In  addition,  a  Force  Development,  Test,  and 
Experiment  (FDTE)  may  be  conducted  to  support  the  development  of 
concepts  and  doctrine,  training,  and  organizations  not 
specifically  tied  to  a  materiel  system  acquisition  (See  Part 
Five)  . 


c.  Continuous  Evaluation  Activities.  The  CBTDEV,  with 
assistance  from  the  independent  operational  evaluator,  should 
evaluate  the  merits  of  a  nonmateriel  solution  to  satisfy  an 
identified  mission  need.  If  CEP  tests  or  FDTEs  are  conducted, 
test  reports  are  to  be  written  and  provided  to  the  CBTDEV.  The 
CBTDEV  should  also  assist  in  the  development  of  any  exit  criteria 
that  may  be  presented  at  MS  I. 

4-3.  Milestone  0  T&E  Requirements 

The  MNS  must  be  developed  and  submitted  to  the  milestone  decision 
authority  for  approval. 


Section  III  .  ... 

T&E  Activities  During  the  Concept  Exploration  and  Definition 

Phase  (Phase  0) 


4-4.  Phase  0  Activities 

A  successful  MS  0  decision  allows  the  program  to  advance  into  the 
Concept  Exploration  and  Development  Phase  (Phase  0) .  Approval  at 
MS  0  allows  for  the  study  of  alternative  concepts  to  meet  the 
need  identified  in  the  MNS.  Phase  0  explores  various  materiel 
alternatives  in  satisfying  the  documented  mission  need. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  4-3. 

(INSERT  FIGURE  4-3  HERE) 

b.  T&E  Activities.  T&E  planning  will  formally  begin  in  this 
phase.  Appropriate  T&E  shall  be  accomplished  and  documented  in 
test  and  evaluation  reports  and  the  TEMP  to  assist  in  selecting 
the  preferred  alternative  system  concept,  associated 
technologies,  and  designs.  In  particular,  the  use  of  modeling 
and  simulation  is  encouraged  in  this  phase  to  aid  in  the 
assessment  of  alternatives  (See  Chapter  16) .  T&E  will  provide 
data  for  concept  evaluation  of  a  potential  requirement,  tactics, 
doctrine,  organization,  training,  transportability,  and  logistic 
support  for  the  preferred  system  concept;  identify  and  assess 
high  risk  areas,  critical  components  and  subsystems;  establish 
safety  for  operational  testing;  and  assess  the  operational  impact 
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Of  the  preferred  concept.  Figure  4-4  illustrates  the  typical  T&E 
planning,  execution,  and  reporting  activities  conducted  during 
this  phase. 

(1)  Planning. 

(a)  The  TIWG  shall  be  established  and  chaired  by  the  PM 
(if  established) ,  or  by  the  appropriate  acquisition  team  until  a 
PM  has  been  established,  and  shall  be  established  upon  receipt  of 
the  approved  MNS  (See  Chapter  8) .  A  draft  Operational 
Requirements  Document  (ORD)  will  be  prepared  and  used  with  the 
System  Threat  Assessment  Report  (STAR)  to  assist  in  developing 
and  finalizing  the  initial  COIC  (See  Part  Three) ,  and  preliminary 
TEMP  (See  Part  Two) .  The  TIWG  will  also  contribute  to  the  T&E 
portions  of  the  AS,  the  RFP  and  other  supporting  documentation 
for  decision  authority  approval  at  MS  I.  Special  efforts  should 
be  made  by  the  TIWG  membership  to  characterize  the  realistic 
environment  of  the  proposed  system,  including  organizational 
structures,  skill  levels,  manpower  requirements,  threat,  mobility 
and  deployability  requirements,  climatic  extremes, 
electromagnetic  environmental  effects,  and  concepts  of  operation 
and  maintenance.  The  acquisition  team  members,  or  PM  (if 
designated) ,  will  also  coordinate  with  the  testers  and 
independent  evaluators  to  optimize  the  use  of  existing  TECOM  test 
facilities  and  initiate  necessary  test  technology  activities. 

(b)  Developmental  and  operational  testing  will  be 
planned  to  provide  data  to  support  evaluations  of  the  system  in 
its  intended  environment.  As  early  as  possible  in  this  phase, 
the  independent  developmental  evaluator  or  assessor  shall  develop 
an  Independent  Evaluation  Plan  (lEP)  (or  Independent  Assessment 
Plan  (lAP))  to  support  the  developmental  evaluation  of  the 
proposed  system  during  this  phase.  Typical  developmental  tests 
include  technical  feasibility  tests,  which  assist  in  determining 
safety  and  the  establishment  of  proposed  system  performance 
specifications.  Developmental  Test  Design  Plans  (TDPs)  will  be 
developed  for  these  tests  by  the  independent  developmental 
evaluator,  and  Detailed  Test  Plans  (DTPs)  will  be  developed  for 
these  tests  by  the  independent  developmental  tester  (See  Part 
Four) .  Typical  operational  tests  may  include  CEP  tests  and  FDTEs 
(See  Part  Five) . 

(2)  Execution.  Technical  feasibility  tests,  CEP  tests, 
and  FDTEs  shall  be  executed  by  the  appropriate  independent 
testers  in  accordance  with  the  approved  test  plans. 

(3)  Reporting.  After  each  developmental  test,  a  test 
report  (TR)  shall  be  written  by  the  independent  developmental 
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tester  and  provided  to  the  independent  developmental  evaluator  or 
assessor  for  inclusion  in  the  Independent  Evaluation  Report  (lER) 
or  Independent  Assessment  Report  (lAR) .  The  independent 
operational  tester  shall  prepare  TRs  for  each  CEP  test  and  FDTE. 
An  Early  Operational  Assessment  (EOA)  or  Abbreviated  Operational 
Assessment  (AOA)  may  be  used  by  the  independent  operational 
evaluator  to  provide  a  status  of  the  system  in  support  of  MS  I 
(See  Parts  Four  and  Five) . 

(INSERT  FIGURE  4-4  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  4-5  contains  the 
CE  activities  to  be  conducted  during  this  phase  (See  Parts  Four 
and  Five) . 


(INSERT  FIGURE  4-5  HERE) 


4-5.  Milestone  I  T&E  Requirements 

Figure  4-6  contains  the  T&E  requirements  to  support  MS  I. 

(INSERT  FIGURE  4-6  HERE) 


Section  IV 

T&E  Activities  During  the  Demonstration  and  Validation  Phase 
(Phase  I) 

4-6.  Phase  I  Activities 

Approval  at  MS  I  establishes  a  new  acquisition  program  and 
Concept  Baseline,  and  authorizes  entry  into  the  Demonstration  and 
Validation  Phase  (Phase  I) .  The  key  objective  of  Phase  I  is  to 
demonstrate  that  the  technologies  critical  to  the  most  promising 
concept  can  be  incorporated  into  the  system  design. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  4-7. 

(INSERT  FIGURE  4-7  HERE) 

b.  T&E  Activities.  T&E  conducted  in  this  phase  includes 
prototyping,  testing,  and  early  operational  assessments  of 
critical  systems,  subsystems,  and  components.  Developmental  T&E 
will  assist  in  the  identification  and  reduction  of  design  risk 
and  indicate  the  degree  to  which  new  or  emerging  technologies 
pose  a  risk  to  the  program.  Operational  T&E  will  assess  the 
degree  to  which  the  selected  design  approach  will  operate  in  the 
intended  operational  environment.  Appropriate  T&E  shall  be 
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accomplished  and  documented  in  test  and  evaluation  reports  and 
the  TEMP.  The  use  of  modeling  and  simulation  is  strongly 
recommended  in  this  phase  to  aid  in  the  assessments  (See  Chapter 
16) .  T&E  will  also  be  conducted  to  address  doctrine,  training, 
organization,  leader  development,  materiel  requirements  and 
logistics  support  aspects  of  the  system  using  surrogate  systems 
if  necessary.  T&E  shall  produce  information  with  which  to 
establish  realistic  program  performance  and  suitability 
thresholds.  Figure  4-8  illustrates  the  typical  T&E  planning, 
execution,  and  reporting  activities  conducted  during  this  phase. 

(1)  Planning. 

(a)  The  TIWG  should  be  expanded  as  necessary  to  include 
the  appropriate  subgroups,  and  interfaces  with  other  working 
groups  should  be  established  (See  Chapter  8).  In  particular,  the 
Live  Fire  Test  and  Evaluation  Working  Group  (LFTEWG)  is  an 
example  of  a  key  working  with  which  the  TIWG  must  interface 
during  this  time.  TIWG  meetings  should  be  held  often,  preferably 
prior  to  the  execution  of  each  test  to  ensure  that  test  details 
are  integrated  and  problems  resolved.  The  update  of  the  COIC  and 
TEMP  to  support  MS  II  can  be  conducted  during  these  TIWGs,  or  at 
specially  designated  TIWGs  (See  Parts  Two  and  Three) .  The  ORD 
and  STAR  will  be  updated,  and  shall  be  used  by  the  TIWG  in  the 
updating  of  the  COIC  and  TEMP.  The  TIWG  can  assist  in  the  update 
of  such  other  documents  as  the  System  MANPRINT  Management  Plan 
(SMMP)  and  the  Integrated  Logistic  Support  Plan  (ILSP) .  The  TIWG 
will  continue  to  contribute  to  the  T&E  portions  of  the  AS,  the 
RFP  and  other  supporting  documentation  for  decision  authority 
approval  at  MS  II.  Sufficient  funds  will  be  programmed  early  by 
the  program  manager  to  ensure  that  adequate  prototypes  ^  and 
ancillary  equipment  and  components  (i.e.;  training  devices, 
ground  support  equipment,  physical  structures,  ammunition  to  test 
systems,  field  maintenance  test  sets,  targets,  simulators, 
stimulators,  models  and  instrumentation)  are  available  and 
adequately  tested.  Outline  Test  Plans  (OTPs)  must  be  developed 
and  participation  in  the  Test  Schedule  and  Review  Committee 
(TSARC)  is  required  if  the  planned  testing  requires  user  troops 
and  resources  (See  AR  15-38) . 

(b)  Developmental  and  operational  testing  will  be 
planned  to  provide  data  to  support  evaluations  of  the  system  in 
its  intended  environment.  As  early  as  possible  in  this  phase, 
the  existing  developmental  lEP  or  lAP  should  be  updated  to 
reflect  information  resulting  from  Phase  0  T&E  activities  and  the 
MS  I  decision  review.  Typical  developmental  tests  include 
engineering  development  tests  (EDTs) ,  which  provide  data  on 
safety,  the  achievability  of  critical  technical  parameters. 
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refinement  and  ruggedization  of  hardware  configurations,  and 
determination  of  technical  risks.  TDPs  will  be  developed  for 
these  tests  by  the  independent  developmental  evaluator,  and  DTPS 
will  be  developed  for  these  tests  by  the  independent 
developmental  tester  (See  Part  Four) .  Typical  operational  tests 
include  Early  User  Test  and  Experiments  (EUTEs)  and,  if 
necessary,  FDTEs.  Operational  Test  Readiness  Reviews  (OTRRs)  and 
Operational  Test  Readiness  Statements  (OTRSs)  are  required  prior 
to  the  start  of  each  EUTE.  The  independent  operational  evaluator 
will  develop  a  Test  and  Evaluation  Plan  (TEP)  for  each  EUTE  in 
this  phase  (See  Part  Five) . 

(2)  Execution.  EDTs,  EUTEs,  and  FDTEs  shall  be  executed  by 
the  appropriate  independent  testers  in  accordance  with  the 
approved  test  plans.  All  required  support  packages  must  be 
developed  and  in  place  before  test  execution  (See  Chapter  9) . 

(3)  Reporting.  After  each  developmental  test,  a  test 
report  (TR)  shall  be  written  by  the  independent  developmental 
tester  and  provided  to  the  independent  developmental  evaluator  or 
assessor  for  inclusion  in  the  Independent  Evaluation  Report  (lER) 
or  Independent  Assessment  Report  (lAR) .  A  TR  will  be  written 
after  the  conduct  of  each  EUTE  and  FDTE  by  the  independent 
operational  tester.  An  EGA,  AOA,  or  Operational  Assessment  (OA) 
will  be  used  by  the  independent  operational  evaluator  to  provide 
a  status  of  the  system  in  support  of  MS  II  (See  Parts  Four  and 
Five) .  MS  II  decisions  to  commit  funds  for  production  long-lead 
items  or  low-rate  initial  production  (LRIP)  must  be  supported  by 
an  EGA,  AOA,  or  OA. 

(INSERT  FIGURE  4-8  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  4-9  contains  the 
CE  activities  to  be  conducted  during  this  phase  (See  Parts  Four 
and  Five) . 

(INSERT  FIGURE  4-9  HERE) 


4-7.  Milestone  II  T&E  Requirements 

Figure  4-10  contains  the  T&E  requirements  to  support  MS  II. 

(INSERT  FIGURE  4-10  HERE) 


Section  V 

T&E  Activities  During  the  Engineering  and  Manufacturing 
Development  Phase  (Phase  II) 
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4-8.  Phase  II  Activities 

Approval  at  MS  II  authorizes  entry  into  the  Engineering  and 
Manufacturing  Development  Phase  (Phase  II) .  The  key  objective  of 
Phase  II  is  to  translate  the  design  approach  developed  in  Phase  I 
into  a  stable,  producible,  and  cost-effective  design. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  4-11. 

(INSERT  FIGURE  4-11  HERE) 

b.  T&E  Activities.  During  this  phase,  the  system  (including 
necessary  training  devices,  threat  simulators,  test  equipment, 
and  computer  resources)  is  engineered,  integrated,  tested, 
evaluated,  and  documented  to  assure  that  the  system  design  is 
stable,  the  system  meets  contract  specifications  and  technical 
parameters,  is  operationally  effective  and  suitable  in  its 
operational  environment,  meets  user  requirements,  and  is  ready 
for  production.  T&E  is  conducted  on  prototype,  production- 
representative,  or  production  systems.  Both  developmental  and 
operational  tests  are  conducted  during  this  phase.  Developmental 
testing  ascertains  whether  engineering  is  complete  (including 
design  and  maintenance  engineering) ,  identifies  design  problems 
and  ascertains  that  solutions  to  these  problems  are  in  hand.  It 
reduces  design  risks,  supports  the  evaluation  of  the  critical 
technical  parameters,  establishes  contractual  compliance, 
provides  information  for  the  type  classification  determination, 
and  validates  general  and  detailed  specifications,  standards,  and 
drawings  for  use  in  production.  Operational  testing  determines 
the  degree  to  which  the  system  is  operationally  effective  and 
suitable.  The  system  design  must  be  sufficiently  mature  to 
provide  adequate  support  packages  for  testing,  and  to  ensure  that 
the  system  tested  is  representative  of  the  production  system  to 
enable  valid  assessments  of  the  system  which  is  expected  to  be 
produced.  If  a  low-rate  initial  production  (LRIP)  decision  was 
made  at  MS  II,  then  this  phase  may  see  the  delivery  of  production 
systems  for  use  in  the  lOT.  Figure  4-12  illustrates  the  typical 
T&E  planning,  executing,  and  reporting  activities  conducted  in 
this  phase. 

(1)  Planning. 

(a)  As  this  phase  is  the  most  test-intensive  phase  of 
the  acquisition  process,  TIWGs  should  be  held  often,  preferably 
prior  to  the  execution  of  each  test  to  ensure  the  test  details 
are  integrated  and  problems  are  resolved  (See  Chapter  8) .  If 
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necessary,  updates  of  the  ORD,  STAR  and  COIC  will  support  the 
TEMP  update,  which  can  be  conducted  during  these  TIWGs,  or  at  a 
specially  designated  TIWG  (See  Parts  Two  and  Three) .  The  TIWG 
should  assist  in  the  update  of  such  other  documents  as  the  System 
MANPRINT  Management  Plan  (SMMP)  and  the  Integrated  Logistic 
Support  Plan  (ILSP) ,  and  contribute  to  the  T&E  portions  of  the  AS 
and  other  supporting  documentation  for  decision  authority 
approval  at  MS  III.  Outline  Test  Plans  (OTPs)  must  be  developed 
and  participation  in  the  Test  Schedule  and  Review  Committee 
(TSARC)  is  required  if  the  planned  testing  requires  user  troops 
and  resources  (See  AR  15-38) . 

(b)  Developmental  and  operational  testing  will  be 
planned  to  provide  data  to  support  evaluations  of  the  system  in 
its  intended  environment.  As  early  as  possible  in  this  phase, 
the  existing  developmental  lEP  or  lAP  should  be  updated  to 
reflect  information  resulting  from  Phase  I  T&E  activities  and  the 
MS  II  decision  review.  Typical  developmental  tests  include 
production-proveout  tests  (PPTs) ,  live-fire  tests  (for  designated 
systems) ,  logistics  demonstrations,  and  the  Preproduction 
Qualification  Test  (PPQT) .  TDPs  will  be  developed  for  each  test 
by  the  independent  developmental  evaluator  or  assessor,  followed 
by  the  DTPs  written  by  the  developmental  tester.  Development 
Test  Readiness  Reviews  (DTRRs)  shall  be  conducted,  and  the  PM 
shall  formally  certify  via  the  Developmental  Test  Readiness 
Statement  (DTRS)  that  the  system  is  ready  for  the  PPQT  to  be 
conducted  (See  Parts  Four  and  Six) .  Typical  operational  tests 
include  Limited  User  Tests  (LUTs) ,  the  Initial  Operational  Test 
(lOT)  (required  for  all  systems) ,  and  the  Follow-On  Operational 
Test  (FOT) ,  if  necessary.  FDTEs  may  also  be  conducted  in  this 
phase.  Except  for  FDTEs,  the  independent  operational  evaluator 
will  develop  a  TEP  for  each  operational  test  in  this  phase,  and 
OTRRs  and  OTRSs  are  required  prior  to  the  start  of  each  test. 

The  Director,  Operational  Test  and  Evaluation  (DOT&E) ,  Office  of 
the  Secretary  of  Defense  (OSD) ,  will  approve  adequacy  of  lOT  test 
plans  for  the  OSD  oversight  systems  prior  to  conduct  of  the  test 
(See  Part  Five) . 

(2)  Execution.  The  PPT  can  consist  of  a  series  of  tests 
on  less  than  system-level  components,  or  on  early  prototypes  of 
the  complete  system.  These  tests  should  be  tailored  to  meet  the 
needs  of  the  specific  program.  The  PPQT  is  the  principal 
developmental  test  in  this  phase,  serving  as  the  final 
developmental  test  prior  to  the  lOT.  The  lOT  must  be  conducted 
on  production  or  production-representative  systems  to  support 
independent  evaluation  of  the  system's  operational  effectiveness 
and  suitability.  The  system  tested  must  be  sufficiently 
representative  of  the  expected  production  system  to  ensure  that 
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T&E  validity  supports  the  production  decision. 

(See  Parts  Four,  Five,  and  Six) . 

(3)  Reporting.  After  each  developmental  test,  a  test 
report  (TR)  shall  be  written  by  the  independent  developmental 
tester  and  provided  to  the  independent  developmental  evaluator  or 
assessor  for  inclusion  in  the  lER  or  lAR.  The  independent 
operational  tester  will  prepare  a  TR  after  conduct  of  the  FDTE. 

A  Test  and  Evaluation  Report  (TER)  will  be  developed  by  the 
independent  operational  evaluator  after  conduct  of  the  lOT  and 
FOT  (if  necessary)  to  provide  a  status  of  the  system  in  support 
of  MS  II.  An  OA  will  be  used  by  the  independent  operational 
evaluator  to  report  on  system  status  at  intermediate  decision 
reviews,  or  where  a  particular  test  is  ongoing  and  results  are 
incomplete  (See  Parts  Four,  Five,  and  Six) . 

(INSERT  FIGURE  4-12  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  4-13  contains 
the  CE  activities  to  be  conducted  during  this  phase  (See  Parts 
Four,  Five,  and  Six) . 

(INSERT  FIGURE  4-13  HERE) 

4-9.  Milestone  III  T&E  Requirements 

Figure  4-14  contains  the  T&E  requirements  to  support  MS  III. 

(INSERT  FIGURE  4-14  HERE) 


Section  VI 

T&E  Activities  During  the  Production  and  Deployment,  and 
Operations  and  Support  Phases 
(Phases  III  and  IV) 


4-10.  Phase  III  and  Phase  IV  Activities 

A  favorable  MS  III  decision  represented  approval  to  build, 
deploy,  and  support  the  system,  and  authorizes  entry  into  the 
Production  and  Deployment  Phase  (Phase  III).  The  key  objective 
of  Phase  III  is  to  establish  a  stable,  efficient  production  and 
support  base,  and  achieve  an  operational  capability  for  the 
system  which  satisfies  the  mission  need.  If  it  is  determined 
that  a  major  modification  or  upgrade  is  warranted  as  a  result  of 
T&E  conducted  in  Phase  III,  a  MS  IV  (Major  Modification  Approval) 
review  will  be  held.  Otherwise,  Phase  III  transitions  smoothly 
into  the  Operations  and  Support  Phase  (Phase  IV)  without  an 
intervening  milestone. 
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a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  these  phases  are  depicted  in  figure  4—15. 

(INSERT  FIGURE  4-15  HERE) 

b.  T&E  Activities 

T&E  shall  be  an  integral  part  of  the  acceptance  and  introduction 
of  system  changes  to  improve  the  system,  react  to  new  threats, 
and  reduce  life— cycle  costs.  Production  qualification  testing 
and  follow-on  operational  testing  will  be  conducted  to  confirm 
and  monitor  performance  and  quality  and  to  verify  the  correction 
of  deficiencies.  These  tests  include  testing  on  the  complete 
system  necessary  to  verify  that  requirements  specified  in 
technical  data  packages  and  production  contracts  for  hardware  or 
software  are  met.  Production  testing  also  provides  a  baseline 
for  follow-on  post  production  testing.  Feedback  of  test  data 
including  sample  data  collection  is  required  to  assess  the 
as-built  quality  of  the  production  items  and  to  determine  the 
need  to  change  test  methodology,  equipment  and  facilities. 

Figure  4-16  illustrates  the  typical  T&E  planning,  executing,  and 
reporting  activities  conducted  in  these  phases. 

(1)  Planning.  TIWG  meetings  should  be  held  often, 
preferably  prior  to  the  execution  of  each  test  to  ensure  the  test 
details  are  integrated  and  problems  are  resolved  (See  Chapter  8) . 
Outline  Test  Plans  (OTPs)  must  be  developed  and  participation  in 
the  Test  Schedule  and  Review  Committee  (TSARC)  is  required  if  the 
planned  testing  requires  user  troops  and  resources  (See  AR  15- 
38) .  Developmental  and  operational  testing  will  be  planned  to 
provide  data  to  support  evaluations  of  the  system  in  its  intended 
environment.  As  early  as  possible  in  Phase  III,  the  existing 
developmental  lEP  or  lAP  should  be  updated  to  reflect  information 
resulting  from  Phase  II  T&E  activities  and  the  MS  III  decision 
review.  Typical  developmental  tests  in  Phase  III  include  the 
Production  Qualification  Test  (PQT) ,  follow-on  production  tests, 
first  article  tests,  comparison  tests,  quality  conformance 
inspections,  and  Post  Deployment  Software  Support  (PDSS)  tests. 
TDPs  will  be  developed  for  each  test  by  the  independent 
developmental  evaluator  or  assessor,  followed  by  the  DTPs  written 
by  the  developmental  tester.  DTRRs  shall  be  conducted,  and  the 
PM  shall  formally  certify  via  the  DTRS  that  the  system  is  ready 
for  the  PQT  to  be  conducted.  Developmental  testing  in  Phase  IV 
consists  of  post-production  testing,  a  follow-on  to  production 
testing,  and  includes  those  surveillance  and  reconditioning  tests 
required  to  measure  the  ability  of  materiel  in  the  field,  in 
storage,  and  after  maintenance  actions  (to  include  repair, 
rebuild,  retrofit,  overhaul,  and  modifications)  to  meet  user 
requirements  (See  Parts  Four  and  Seven).  The  typical  operational 
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test  conducted  in  Phase  III  is  the  FOT,  if  necessary.  The 
independent  operational  evaluator  will  develop  a  TEP  for  the  FOT 
and  OTRRs  and  OTRSs  are  required  prior  to  the  start  of  the  FOT. 
There  is  no  required  operational  testing  during  Phase  IV  however, 
the  results  of  Army  Training  and  Evaluation  Programs  (ARTEP) , 
Field  Training  Exercises  (FTX) ,  reforgers,  and  Sample  Data 
Collection  (SDC)  are  all  sources  of  information  from  which  the 
independent  operational  evaluator  can  continue  to  periodically 
monitor  the  systems  continuing  ability  to  meet  the  identified 
mission  need  (See  Part  Five) . 


(2)  Execution.  The  PQT  is  the  principal  development  test 
in  Phase  III.  It  is  a  system-level  test  to  verify  that  the 
production  system  meets  contract  specifications  and  technical 
parameters,  to  determine  the  adequacy  and  timeliness  of  any 
corrective  actions  indicated  by  previous  testing,  and  to  validate 
the  manufacturer's  facilities,  procedures,  and  processes.  The 
FOT  shall  be  conducted  as  necessary  to  ensure  that  the  production 
version  of  the  system  performs  as  well  as  reported  at  MS  III, 
demonstrates  expected  performance  and  reliability  improvement, 
evidences  correction  of  deficiencies  identified  during  earlier 
tests,  ensures  that  new  problems  have  not  been  injected  by  the 
production  process,  and  determines  overall  readiness  of  the 
system  to  be  fielded.  All  testing  in  Phase  III  is  focused  on 
confirming  fixes  to  problems  identified  in  earlier  tests  (See 
Parts  Four,  Five,  and  Seven) . 

(3)  Reporting.  After  each  developmental  test  in  Phase 
III,  a  TR  shall  be  written  by  the  independent  developmental 
tester  and  provided  to  the  independent  developmental  evaluator  or 
assessor  for  inclusion  in  the  lER  or  lAR.  A  TER  will  be 
developed  by  the  independent  operational  evaluator  to  provide  a 
status  of  the  system  resulting  from  the  FOT.  An  OA  will  be  used 
by  the  independent  operational  evaluator  to  report  on  system 
status  at  intermediate  decision  reviews,  or  if  the  FOT  is  ongoing 
and  results  are  incomplete  (See  Parts  Four,  Five,  and  Seven) . 


(INSERT  FIGURE  4-16  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  4-17  contains 
the  CE  activities  to  be  conducted  during  Phase  III  and  Phase  IV 
(See  Parts  Four,  Five,  and  Seven) . 


(INSERT  FIGURE  4-17  HERE) 
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4-11.  Milestone  IV  T&E  Requirements 

A  MS  IV,  Major  Modification  Approval,  review  is  required  only  if 
major  upgrades  to  the  system  currently  in  production  are 
warranted.  This  need  may  be  brought  about  by  a  change  in  the 
system's  threat,  a  major  deficiency  identified  during  FOT  or 
operational  training  and  support,  or  by  an  opportunity  to  reduce 
the  cost  of  ownership.  If  a  major  modification  program  is 
approved,  the  milestone  decision  authority  will  determine  which 
acquisition  phase  the  program  should  enter  (See  DODI  5000.2). 
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FIGURE  4-1 

Life  Cycle  Management  Model 
For  Materiel  Systems 


DETERMINATION  OF  MISSION  NEEDS 
KEY  ACTIVITIES 


•  Identification  of  mission  deficiencies  or  improvement 
opportunites . 

•  Evaluation  of  nonmateriel  solutions  to  satisfy  mission 
deficiencies. 

•  Development  of  MNS  if  nonmateriel  solutions  are  not 
feasible. 


igure  4-2 
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CONCEPT  EXPLORATION  AND  DEVELOPMENT 
Phase  0 

KEY  ACQUISITION  ACTIVITIES 

•  Definition  and  evaluation  of  the  feasibility  of  alternative 
concepts . 

•  Definition  of  the  most  promising  system  concept. 

•  Establish  a  proposed  Concept  Baseline. 

•  Development  of  a  proposed  acquisition  strategy  for  the  most 
promising  concept. 

•  Development  of  key  system  characteristics  and  operational 
constraints. 

•  Development  of  proposed  program-specific  exit  criteria  that 
must  be  accomplished  during  Phase  I. 


Figure  4-3. 


L4-|4> 


H-n 


Figure  4-4  Phase  O.  Concept  Exploration  and  Definition 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
DURING  CONCEPT  EXPLORATION/ DEFINITION 
Phase  0 


•  Participate  in  the  TIWG. 

•  Assist  in  selecting  the  preferred  alternative  to  resolve 
mission  area  deficiencies. 

•  Participate  in  ORD  development  efforts. 

•  Support  the  initial  COIC  development  and  approval  process. 

•  Assist  in  developing  system  characteristics  and  exit 
criteria. 

•  Participate  in  development,  staffing,  and  approval  of  the 
TEMP. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests. 

•  Develop  and  provide  developmental  lER  or  lAR  to  appropriate 
decision  makers. 

•  Develop  and  provide  EOA  or  AOA  to  support  MS  I  decision. 
Figure  4-5. 
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MILESTONE  I  T&E  REQUIREMENTS 


•  Draft  ORD. 

•  Approved  initial  COIC. 

•  Preliminary  TEMP. 

•  Developmental  lER  or  lAR;  EOA  or  AOA. 


DEMONSTRATION  AND  VALIDATION 
Phase  I 

KEY  ACQUISITION  ACTIVITIES 


•  Better  define  the  critical  design  characteristics  and 
expected  capabilities  of  the  system  concept. 

•  Refinement  of  the  AS. 

•  Establish  a  proposed  Development  Baseline. 

•  Development  of  proposed  program-specific  exit  criteria  that 
must  be  accomplished  during  Phase  II. 

•  Determine  plan  for  committing  to  low-rate  initial 
production. 

Figure  4-7. 
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Rgure  4-8  Phase  I  Demonstration  and  Validation 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
DEMONSTRATION  AND  VALIDATION 
Phase  I 


•  Continued  participation  in  the  TIWG. 

•  Support  the  COIC  update  and  approval  process. 

•  Support  the  ORD  update  and  approval  process. 

•  Participate  in  the  update,  staffing,  and  approval  of  the 
TEMP. 

•  Support  COEA  update  efforts. 

•  Assist  in  the  development  of  exit  criteria. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests. 

•  Develop  and  provide  developmental  lER  or  lAR  to  appropriate 
decision  makers. 

•  Develop  and  provide  EGA,  AOA,  or  OA  to  support  intermediate 
decision  and  MS  II  decision. 

Figure  4-9. 
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MILESTONE  II  T&E  REQUIREMENTS 


•  Updated  ORD. 

•  Updated  COIC. 

•  Updated  TEMP. 

•  Developmental  lER  or  lAR;  EOA,  AOA,  or  OA. 
figure  4-10. 


ENGINEERING  AND  MANUFACTURING 
DEVELOPMENT 
Phase  II 

KEY  ACQUISITION  ACTIVITIES 


•  Validate  the  manufacturing  and  production  process. 

•  Refinement  of  the  AS. 

•  Establish  a  proposed  Production  Baseline. 

•  Establish  system  configuration  baseline. 

•  Demonstrate  through  testing  that  system  capabilities  meet 
contract  specifications,  satisfies  the  mission  need,  and  meets 
the  minimum  acceptable  operational  performance  requirements. 

•  Demonstrate  that  the  low-rate  initial  production  provides 
assurance  that  the  design  is  stable  and  capable  of  being 
produced  efficiently. 

•  Development  of  proposed  program-specific  exit  criteria  that 
must  be  accomplished  during  Phase  III,  if  appropriate. 


Figure  4-11 


ENGINEERINQ  AND  MANUFACTURING  DEVELOPMENT 
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Figure  4-12  Phase  11,  Engineering  and  Manufacturing  Development 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
ENGINEERING  AND  MANUFACTURING  DEVELOPMENT 

Phase  II 


•  Continued  participation  in  the  TIWG. 

•  Support  the  COIC  update  and  approval  process. 

•  Support  the  ORD  update  and  approval  process  (if 
appropriate) . 

•  Participate  in  the  update,  staffing,  and  approval  of  the 
TEMP. 

•  Support  COEA  update  efforts. 

•  Assist  in  the  development  of  exit  criteria,  if  appropriate. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests. 

•  Develop  and  provide  developmental  lER  or  lAR  to  appropriate 
decision  makers. 

•  Develop  and  provide  operational  TER  to  support  MS  III 
decision,  and  OAs  to  support  intermediate  decision  reviews. 

Figure  4-13 
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MILESTONE  III  T&E  REQUIREMENTS 

•  Updated  ORD,  if  appropriate. 

•  Updated  COIC. 

•  Updated  TEMP. 

•  Developmental  lER  or  lAR;  Operational  TER  and  OA. 
i^^igure  4-14. 


PRODUCTION  AND  DEPLOYMENT 
Phase  III 

OPERATIONS  AND  SUPPORT 
Phase  IV 

KEY  ACQUISITION  ACTIVITIES 


•  Update  configuration  baseline. 

•  Refinement  of  cost  information. 

•  Through  testing,  confirm  and  monitor  performance  and  quality 
and  verify  correction  of  def ieciencies. 

•  Ensure  the  fielded  system  continues  to  provide  the 
capabilities  required  to  meet  the  identified  mission  need. 

•  Identify  the  need  for  major  upgrades  to  the  system  currently 
in  production  that  require  a  MS  IV,  Major  Modification 
Approval,  review. 


Figure  4-15 


Figure  4-16  Phase  lit.  Production  and  Deployment/Phase  IV,  Operations  and  Support 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
PRODUCTION  AND  DEPLOYMENT 
Phase  III 

OPERATIONS  AND  SUPPORT 
Phase  IV 


•  Continued  participation  in  che  TIWG. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests . 

•  Develop  and  provide  developmental  lER  or  lAR  to  appropriate 
decision  makers. 

•  Develop  and  provide  operational  TER  and  OAs  to  support 
intermediate  decision  reviews. 

•  Assist  in  the  identification  of  deficiencies  which  may 
warrant  a  MS  IV  (Major  Modification  Approval)  review. 


Figure  4-17 
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Chapter  5 

Test  and  Evaluation  in  Support  of  the  Information  Mission  Area 
Acquisition  Process 


Section  I 
Introduction 


5-1.  General 

a.  T&E  is  an  essential  activity  in  support  of  the  acquisition 
of  information  systems,  systems  that  evolve,  are  acquired,  or  are 
developed  which  incorporate  information  technology.  These 
information  systems  belong  to  the  five  Information  Mission  Area 
(IMA)  disciplines  discussed  in  AR  25-3.  T&E  in  support  of  the 
IMA  acquisition  process  plays  a  key  role  in  the  life  cycle  of 
information  systems,  providing  data  to  assist  in  their  selection, 
development,  acquisition,  use,  maintenance  and  support.  T&E  is 
inherent  in  the  activities  that  provide  new  information 
technologies  to  be  exploited;  it  is  used  to  support  the  selection 
of  best  solutions  to  satisfy  an  IMA  deficiency;  and  it  verifies 
that  the  Army  is  designing,  developing,  producing,  and 
maintaining  information  systems  that  satisfy  the  users'  needs. 

b.  Comprehensive  developmental  and  operational  testing  shall 
be  conducted  on  all  information  systems.  Early  detailed  software 
T&E  planning  is  critical  to  meaningful  evaluations  and 
assessments,  as  well  as  to  the  successful  development  of  the 
system.  The  T&E  strategy  shall  specify  the  impact  on  risk  of  the 
technologies  and  processes  selected  for  system  development  during 
the  entire  life  cycle  of  the  system.  Test  methodologies  shall 
include  realistic  software  test  environments  and  scenarios. 

c.  Developmental  T&E  shall  be  planned  and  incorporated  into 
the  information  system's  development  process  to  verify 
conformance  to  technical  specifications  and  performance 
attributes  to  technical  objectives  and  requirements. 

Developmental  T&E  shall  encompass  the  system  hardware,  software, 
code  documentation,  users  manuals,  training  material,  interfaces, 
compatibility,  and  interoperability  with  existing  or  planned 
systems.  Operational  T&E  shall  examine  system  effectiveness  and 
suitability  under  operationally  realistic  conditions  when  the 
system  is  operated  by  typical  users.  In  addition,  the  T&E  should 
address  the  system's  compatibility  and  interoperability  with 
users  and  other  systems. 

d.  Continuous  evaluation  (CE)  as  discussed  in  Chapter  2  is  a 
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major  ingredient  in  the  T&E  which  supports  the 

process.  It  should  begin  as  early  as  the  Project  Management  Plan 
(PMP)  process  and  continue  through  the  system's  postdeployment 
activities. 


e.  Detailed  information  concerning  software  T&E  of  IMA 
systems  is  contained  in  Part  Seven.  Roles  and  procedures  for  all 
segments  of  software  T&E,  including  the  maintenance  of  software 
after  deployment,  are  discussed. 


f.  The  phases  and  milestones  for  the  life-cycle  system 
management  model  (LCSMM)  for  information  systems  is  illustrated 
in  figure  5-1. 


(INSERT  FIGURE  5-1  HERE) 


Section  II 

T&E  Activities  During  the  Need  Justification  Phase 
(Phase  0) 


5-2.  Phase  0  Activities  .  -i-H-io 

Phase  0  defines  and  documents  a  mission  need  and  validates  this 
need.  The  phase  begins  when  a  mission  deficiency  is  identified 
or  an  opportunity  is  recognized  to  improve  mission  performance. 
This  phase  ends  at  MS  0,  which  formally  approves  the  Mission  Need 
Statement  (MNS) . 

a.  Key  Acquisition  Events.  In  this  phase,  the  functional 
proponent  (FP)  determines  whether  a  mission  deficiency  or  an 
opportunity  to  improve  an  information  system  is  important  enough 
to  warrant  further  analysis  and  development  of  a  system.  The  FP 
ensures  that  proper  planning  and  evaluation  are  successfully 
carried  out.  The  key  acquisition  activites  conducted  during  this 
phase  are  depicted  in  figure  5-2. 

(INSERT  FIGURE  5-2  HERE) 


b.  T&E  Activities.  T&E  is  usually  not  conducted  until  after 
the  MS  I  decision.  However,  in  those  cases  where  T&E  may  be  ^ 
applicable,  T&E  generally  consists  of  demonstrations  to  assist  in 
the  identification  of  mission  deficiencies;  evaluation  of  the 
impact  of  the  deficiencies  on  the  performance  of  the  mission;  and 
evaluation  of  the  impact  of  essential  functional  and  technical 
constraints  affecting  potential  alternative  solutions. 


c.  Continuous  Evaluation  Activities.  If  applicable,  CE 
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activities  during  this  phase  consist  of  an  evaluation  of  the 
impact  of  deficiencies  on  the  performance  of  the  mission. 

5-3.  Milestone  0  T&E  Requirements  ,  , 

The  MNS  must  be  developed  and  submitted  to  the  milestone  decision 
authority  for  approval. 


Section  III 

T&E  Activities'  During  the  Concepts  Development  Phase  (Phase  I) 


5-4.  Phase  I  Activities 

A  successful  MS  0  decision  allows  the  program  to  advance  into  the 
Concepts  Development  Phase  (Phase  I) .  Phase  I  identifies  and 
evaluates  alternative  functional  and  technical  concepts  that 
satisfy  the  approved  MNS,  and,  based  on  the  results  of  these 
evaluations,  selects  the  best  functional  or  technical  concept. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  5-3. 

(INSERT  FIGURE  5-3  HERE) 

b.  T&E  Activities.  T&E  will  support  the  evaluation  of 
alternative  concepts  that  satisfy  the  approved  MNS  and  support 
the  selection  of  the  best  functional  or  technical  concept. 

Initial  planning  for  T&E  shall  begin  in  this  phase,  including  the 
establishment  of  requirements  for  independent  T&E  and  quality 
assurance  programs.  Modeling  and  simulation,  rapid  prototyping, 
and  any  other  techniques  shall  be  considered  to  reduce  program 
risks  and  future  costs.  Metrics  for  cost  and  schedule  shall  be 
developed  and  integrated  into  the  T&E  strategy.  Figure  5-4 
illustrates  the  typical  planning,  execution,  and  reporting 
activities  conducted  during  this  phase. 

(1)  Planning.  The  PM  shall  establish  the  TIWG  during  this 
phase  (See  Chapter  8).  The  UFD  will  be  developed  and  used  in 
conjunction  with  the  MNS  to  develop  and  finalize  initial  COIC 
(See  Part  Three) .  The  preliminary  TEMP  will  also  be  developed  by 
the  TIWG  (See  Part  Two) .  T&E  planning  will  be  incorporated  in 
the  acquisition  strategy,  the  RFP,  the  SDP,  and  other  supporting 
documentation  for  the  milestone  decision  authority  for  MS  I. 

(2)  Execution.  Typically  no  developmental  or  operational 
testing  is  conducted  during  this  phase. 

(3)  Reporting.  If  appropriate,  an  EGA  may  be  required  to 
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assess  the  potential  of  the  selected  concept  with  respect  to 
operational  effectiveness  and  suitability  (See  Parts  Five  and 
Seven)  . 


(INSERT  FIGURE  5-4  HERE) 


c.  Continuous  Evaluation  Activities.  Figure  5-5  contains  the 
CE  activities  to  be  conducted  during  this  phase  (See  Part  Seven) . 

(INSERT  FIGURE  5-5  HERE) 

5-5.  Milestone  I  T&E  Requirements 

Figure  5-6  contains  the  T&E  requirements  to  support  MS  I. 

(INSERT  FIGURE  5-6  HERE) 


Section  IV 

T&E  Activities  During  the  Design  Phase  (Phase  II) 


5-6.  Phase  II  Activities 

A  successful  MS  I  decision  allows  the  program  to  advance  into  the 
Design  Phase  (Phase  II) .  The  purpose  of  Phase  II  is  to  complete 
the  technical  specifications  of  the  information  system  and  to 
validate  the  selected  system  design. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  5—7 . 

(INSERT  FIGURE  5-7  HERE) 

b.  T&E  Activities.  T&E  will  support  the  completion  of  the 
technical  specifications  and  support  those  remaining 
demonstration  and  prototyping  activities.  Adequate  T&E  shall  be 
accomplished  to  complete  the  identification  of  the  technical 
risks  associated  with  the  selected  design,  and  shall  establish 
realistic  system  performance  and  suitability  thresholds. 

Modeling,  simulation,  and  prototyping  are  encouraged  to  support 
an  EGA  prior  to  MS  II.  In  addition  to  the  metrics  developed  in 
the  previous  phase,  metrics  for  computer  resource  utilization 
(CRU) ,  software  engineering  environment  (SEE) ,  requirements 
traceability,  and  requirements  stability  shall  be  developed  and 
integrated  into  the  T&E  strategy.  Figure  5-8  illustrates  the 
typical  T&E  planning,  execution,  and  reporting  activities 
conducted  during  this  phase. 
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(1)  Planning.  TIWG  meetings  should  be  held  as  required  to 
continue  planning  for  developmental  and  operational  T&E,  and  to 
update  the  TEMP  in  support  of  MS  II  (See  Part  Two) .  The  FD  will 
be  finalized  and  used  to  update  the  COIC  (See  Part  Three) . 

(2)  Execution.  Typically  no  developmental  or  operational 
testing  is  conducted  during  this  phase. 

(3)  Reporting.  An  EOA,  based  on  demonstrations,  modeling, 
simulation,  and  other  analytical  techniques,  will  be  provided  by 
the  independent  operational  evaluator  in  support  of  the  MS  II 
decision  review  (See  Parts  Five  and  Seven) . 

(INSERT  FIGURE  5-8  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  5-9  contains  the 
CE  activities  to  be  conducted  during  this  phase  (See  Part  Seven) . 

(INSERT  FIGURE  5-9  HERE) 

5-7.  Milestone  II  T&E  Requirements 

Figure  5-10  contains  the  T&E  requirements  to  support  MS  II. 

(INSERT  FIGURE  5-10  HERE) 


Section  V 

T&E  Activities  During  the  Development  Phase  (Phase  III) 


5-8.  Phase  III  Activities 

A  successful  MS  II  decision  allows  the  program  to  advance  into 
the  Development  Phase  (Phase  III) .  The  purpose  of  Phase  II  is  to 
develop  the  information  system,  test  the  total  system  to  ensure 
it  satisfies  the  user's  requirements,  and  to  prepare  the 
information  system  for  deployment. 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  this  phase  are  depicted  in  figure  5-11. 

(INSERT  FIGURE  5-11  HERE) 

b.  T&E  Activities.  T&E  will  be  conducted  during  this  phase 
on  completed  prototype,  preproduction,  or  production- 
representative  information  systems.  The  T&E  will  include  all 
testing  required  to  confirm  that  all  deficiencies  have  been 
identified;  that  solutions  to  these  problems  are  available;  and 
to  provide  a  valid  estimate  of  the  system's  safety,  operational 
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effectiveness  and  suitability.  The  system  design  must  be 
sufficiently  mature  so  that  the  tested  system  will  provide  data 
to  accurately  assess  the  production  system.  All  remaining 
required  software  metrics  shall  be  developed  and  integrated  into 
the  T&E  strategy.  Figure  5-12  illustrates  the  typical  T&E 
planning,  execution,  and  reporting  activities  conducted  during 
this  phase. 

(1)  Planning.  As  early  as  possible  in  this  phase,  the 
independent  developmental  evaluator  shall  develop  an  lEP  to 
support  the  developmental  evaluation  of  the  system  in  this  phase. 
Typical  developmental  tests  include  software  development  tests 
(SDTs) ,  which  consist  of  program  or  module  and  cycle  or  systems 
levels  of  testing,  and  the  software  qualification  test.  Unit  or 
module  tests  are  executed  on  local  testbed  hardware  using 
benchmark  test  files.  Cycle  or  system  tests  involve  testing  the 
combination  of  linkage  of  programs  or  modules  into  major 
processes.  The  software  qualification  test  is  a  total  system 
test  conducted  by  the  independent  developmental  tester  using  live 
data  files  supplemented  with  user-prepared  data  and  executed  on 
target  hardware.  A  TDP  will  be  developed  for  the  software 
qualification  test,  followed  by  the  DTP,  written  by  the 
developmental  tester.  DTRRs  and  DTRSs  are  required  prior  to 
execution  of  the  software  qualification  test  (See  Part  Seven) . 
Typical  operational  tests  include  the  LUT,  lOT,  and  FOT  (if 
necessary) .  Each  system  shall  undergo  an  lOT  in  at  least  one 
representative  site  using  real  functional  data.  The  independent 
operational  evaluator  will  develop  a  TEP  for  each  operational 
test  in  this  phase.  OTRRs  and  OTRSs  are  required  to  verify  that 
the  system  is  production-representative  and  ready  for  the  lOT  or 
FOT  to  be  conducted  (See  Parts  Five  and  Seven) .  The  DOT&E  will 
approve  the  adequacy  of  the  lOT  TEP  for  OSD  MAISRC  systems  prior 
to  the  conduct  of  the  test  (See  Part  Five) .  TIWG  meetings  should 
be  held  as  required  to  continue  planning  for  developmental  and 
operational  T&E  (See  Chapter  8) .  Updates  of  the  COIC  and  TEMP  to 
support  the  MS  III  decision  must  also  be  conducted  (See  Parts  Two 
and  Three) .  OTPs  must  be  developed  and  participation  in  the 
TSARC  is  required  if  the  planned  testing  requires  user  troops  and 
resources  (See  AR  15-38) . 

(2)  Execution.  The  system  qualification  test  is  the 
principal  developmental  test  in  this  phase,  serving  as  the  final 
developmental  test  prior  to  the  lOT.  The  PM  shall  formally 
certify  via  the  DTRS  that  the  system  is  ready  for  the  system 
qualification  test  to  be  conducted.  The  lOT  may  include  the 
procedures  of  what  was  formerly  known  as  the  software  acceptance 
test.  The  lOT  must  be  conducted  on  production-representative 
systems  to  support  independent  evaluation  of  the  system's 
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operational  effectiveness  and  suitability.  The  system  tested 
must  be  sufficiently  representative  of  the  expected  production 
system  to  ensure  that  T&E  validity  supports  the  production 
decision.  (see  Parts  Five  and  Seven) . 

(3)  Reporting.  A  TR  shall  be  written  by  the  independent 
developmental  tester  after  the  software  qualification  test  and 
provided  to  the  independent  developmental  evaluator  for  input 
into  the  lER.  A  TER  shall  be  written  by  the  independent 
operational  evaluator  after  the  lOT  and  FOT.  An  OA  will  be 
written  if  testing  is  incomplete  or  a  review  prior  to  MS  III  is 
requested.  Before  the  MS  III  review  the  results  of  testing  (to 
include  all  system  testing  and  preproduction  qualification 
testing)  of  production  or  production-representative  articles 
shall  confirm  that  all  deficiencies  have  been  identified;  that 
solutions  to  these  problems  are  available;  and  that  the  items  or 
components  actually  tested  are  effective  and  suitable  for  their 
intended  use  (See  Parts  Five  and  Seven) . 

(INSERT  FIGURE  5-12  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  5-13  contains 
the  CE  activities  to  be  conducted  during  this  phase  (See  Part 
Seven) . 


(INSERT  FIGURE  5-13  HERE) 

5-9.  Milestone  III  T&E  Requirements 

Figure  5-14  contains  the  T&E  requirements  to  support  MS  III. 

(INSERT  FIGURE  5-14  HERE) 


Section  VI 

T&E  Activities  During  the  Deployment  and  Operations  Phases 
(Phases  IV  and  V) 

5-10.  Phase  IV  and  Phase  V  Activities 

A  successful  MS  III  decision  allows  the  program  to  advance  into 
the  Deployment  Phase  (Phase  IV) .  The  purpose  of  Phase  IV  is  to 
field  the  information  system  in  accordance  with  the  approved 
deployment  plan.  Transition  into  the  Operations  Phase  (Phase  V) 
occurs  when  progam  control  is  passed  from  the  PM/MATDEV  to  the 
Operations  Manager.  There  is  no  milestone  required  for  this 
action.  Phase  V  objectives  are  to  operate  and  maintain  the 
system,  evaluate  its  effectiveness  and  benefits,  implement  the 
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short-term  postdeployment  modernization  plan,  and  plan  for  long¬ 
term  modernization.  MS  IV  occurs  at  the  end  of  the  Operations 
Phase  (Phase  V) . 

a.  Key  Acquisition  Events.  The  key  acquisition  activities 
conducted  during  these  phases  are  depicted  in  figure  5-15. 

(INSERT  FIGURE  5-15  HERE) 

b.  T&E  Activities.  T&E  will  support  the  fielding,  operation, 
and  maintenance  of  the  information  system;  support  the  evaluation 
of  the  effectiveness  and  benefits  of  the  system;  support  the 
implementation  of  the  short-term  postdeployment  modernization 
plan;  and  support  the  long-term  existing  modernization.  TIWG 
meetings  should  be  held  as  required  to  continue  planning  for 
developmental  and  operational  T&E  (See  Chapter  8) .  The  principal 
T&E  efforts  will  be  PDSS  tests,  which  consist  of  modifications 
and  maintenance  of  software  in  deployed  systems,  the  FOT,  and,  as 
required,  Supplemental  Site  Tests  (SSTs) .  As  early  as  possible, 
the  independent  developmental  evaluator  shall  update  the  lEP  to 
reflect  information  resulting  from  Phase  III  T&E  activities  and 
the  MS  III  decision  review.  A  TDP  will  be  developed  for  each 
PDSS  test,  followed  by  the  DTP,  written  by  the  developmental 
tester  (See  Part  Seven) .  The  independent  operational  evaluator 
will  develop  a  TEP  for  the  FOT.  DTRRs  and  DTRSs  are  required 
prior  to  execution  of  a  PDSS  test;  OTRRs  and  OTRSs  are  required 
prior  to  execution  of  the  FOT.  A  TR  shall  be  written  by  the 
independent  developmental  tester  after  the  PDSS  test  and  provided 
to  the  independent  developmental  evaluator  for  input  into  the 
lER.  A  TER  shall  be  written  by  the  independent  operational 
evaluator  after  the  FOT,  or  an  OA  will  be  written  if  testing  is 
incomplete  or  an  intermediate  review  is  requested  (See  Parts  Five 
and  Seven) .  Figure  5-16  illustrates  the  typical  T&E  planning, 
execution,  and  reporting  activities  conducted  during  this  phase. 

(INSERT  FIGURE  5-16  HERE) 

c.  Continuous  Evaluation  Activities.  Figure  5-17  contains 
the  CE  activities  to  be  conducted  during  these  phases  (See  Part 
Seven) . 

(INSERT  FIGURE  5-17  HERE) 

5-11.  Milestone  IV  T&E  Requirements 

MS  IV  T&E  requirements  include  the  developmental  lER,  the 
operational  TER,  and  any  operational  OAs. 
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Section  VII 

T&E  Activities  During  the  Revalidation  Phase  (Phase  VI) 


5-12.  Phase  VI  and  MS  V  Activities 

The  purpose  of  the  Revalidation  Phase  (Phase  VI)  is  to  determine 
whether  the  information  system  conforms  to  architectural 
requirements  and  continues  to  satisfy  mission  needs,  or  should  be 
terminated.  The  activities  undertaken  by  the  FP  and  Operations 
Manager  include  ensuring  that  all  class  II  through  V  programs 
receive  a  MS  V  Revalidation  review  by  the  appropriate  approving 
authority.  If  a  revalidation  review  results  in  approval  to 
implement  system  modernization,  the  approval  authority  will 
determine  which  acquisition  phase  the  system  will  enter.  No  T&L 
activities  are  usually  required  during  this  phase,  however  all 
participants  in  the  CE  process  should  monitor  the  information 
system  with  respect  to  potential  implementation  of  modernizations 
(See  Part  Seven) . 
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For  Information  Systems 


NEED  JUSTIFICATION 
Phase  0 

KEY  ACQUISITION  ACTIVITIES 

•  Identification  of  mission  deficiencies  or  improvement 
opportunites . 

•  Identification  of  essential  functional,  technical,  and 
financial  constraints  and  assumptions  which  affect  potential 
alternative  solutions. 

•  Integration  of  the  results  of  these  activities  into  the  MNS. 


Figure  5-2 


CONCEPTS  DEVELOPMENT 
Phase  I 

KEY  ACQUISITION  ACTIVITIES 


•  Assessments  of  alternative  functional  and  technical 
concepts . 

•  Selection  of  the  best  functional  or  technical  concept  to 
satisfy  the  mission  need. 

•  Evaluation  and  selection  of  the  appropriate  acquisition 
strategy  to  implement  the  recommended  program. 

•  Initial  planning  for  the  design,  development,  testing, 
deployment,  training,  maintenance,  and  modernization  (if 
appropriate)  of  the  proposed  information  system. 

•  Development  of  the  Configuration  Management  Plan  (CMP) . 


Figure  5-3 


Figure  5-4  Phase  I,  Concepts  Development 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
CONCEPTS  DEVELOPMENT  PHASE 
Phase  I 


•  Determine  if  the  system  requirements  are  testable,  and  that 
measurable  criteria  are  established. 

•  Ensure  that  user  requirements  are  traceable  to  the  system 
specif ications . 

•  Ensure  that  the  cost  and  schedule  software  metrics  are 
properly  developed  and  incorporated  into  the  T&E  strategy. 

•  Develop  and  provide  EOA  to  appropriate  decision  makers. 


Figure  5-5 
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MILESTONE  I  T&E  REQUIREMENTS 

•  Updated  and  revalidated  MNS. 

•  Validated  UFD,  and  draft  FD. 

•  Initial  COIC. 

•  Preliminary  TEMP. 

Figure  5-6 
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DESIGN 
Phase  II 

KEY  ACQUISITION  ACTIVITIES 


•  Ensuring  that  the  system  design  is  based  on  functional 
requirements,  including  the  FD. 

•  Integration  of  results  of  remaining  demonstrations  and 
prototyping  activities  into  the  system  design. 

•  Selection  of  modern  development  technologies  to  be  used  in 
system  development. 

•  Development  of  a  product  baseline  in  accordance  with  the 
CMP. 


Figure  5-7 
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Figure  5-8  Phase  II.  Design 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 

DESIGN  PHASE 
Phase  II 


•  Ensure  that  the  requirements  in  the  UED,  FD,  and  MNS  are 
traceable  to  the  system  specification  and  among  each  other. 

•  Ensure  that  performance  and  suitability  thresholds  have  been 
properly  determined  and  are  reflected  in  the  TEMP. 

•  Ensure  that  the  required  software  metrics  are  properly 
developed  and  integrated  into  the  T&E  strategy. 

•  Develop  and  provide  EOA  to  appropriate  decision  makers. 


Figure  5-9 
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MILESTONE  II  T&E  REQUIREMENTS 


•  Finalized  and  validated  FD. 

•  Updated  COIC. 

•  Updated  TEMP. 

•  EOA  provided  by  the  independent  operational  evaluator. 
Figure  5-10 


DEVELOPMENT 
Phase  III 

KEY  ACQUISITION  ACTIVITIES 


•  Full-scale  information  system  development. 

•  Conducting  developmental  and  operational  testing  to  validate 
that  the  system  design  is  stable  and  that  it  meets  user 
functional  requirements. 

•  Validation  that  the  information  system  is  ready  for 
peacetime,  mobilization,  and  wartime  operational  use. 

•  Planning  for  deployment,  training,  operations,  maintenance, 
and  logistic  support  of  the  information  system. 


Figure  5-11 


Rgure  5-12  Phase  Ml,  Development 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
DEVELOPMENT  PHASE 
Phase  III 


•  Continue  participation  in  the  TIWG  and  TEMP  update  process. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests. 

•  Ensure  that  all  required  software  metrics  are  properly 
developed  and  integrated  into  the  T&E  strategy. 

•  Develop  and  provide  developmental  lER  to  appropriate 
decision  makers. 

•  Develop  and  provide  OA  to  support  intermediate  decisions. 

•  Develop  and  provide  TER  to  support  MS  III  decision. 


Figure  5-13 
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DEPLOYMENT  AND  OPERATIONS 
Phase  IV  and  Phase  V 
KEY  ACQUISITION  ACTIVITIES 


•  Transition  planning  from  the  PM/MATDEV  to  the  information 
system  Operations  Manager. 

•  Availability  of  resources  to  satisy  the  requirements  of  the 
proposed  deployment  schedule  and  full  operations  and 
maintenance. 

•  Postdeployment  operational  assessment  planning  for  the  MS  IV 
decision  review. 

•  Planning  for  existing  system  modernization  assessment. 

•  Effective  operating  procedures  are  developed  to  evaluate 
benefits,  correct  malfunctions,  and  respond  to  user  needs. 


CONTINUOUS  EVALUATION  (CE)  ACTIVITIES  DURING 
DEPLOYMENT  AND  OPERATIONS 
Phase  IV  and  Phase  V 


•  Continue  participation  in  the  TIWG. 

•  Plan  and  execute  all  required  developmental  and  operational 
tests. 

•  Develop  and  provide  developmental  lER  to  appropriate 
decision  makers. 

•  Develop  and  provide  operational  TER  and  OAs  to  support 
intermediate  decision  reviews. 


Figure  5-17 
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Chapter  6 

T&E  for  Tailored  Acquisition 


Section  I 
Introduction 


6-1.  General 

The  Army  often  uses  expedited  acquisition  processes  to  reduce  the 
acquisition  lead  time.  The  Army  embarks  on  these  acquisitions 
through  two  motivations: 

a.  To  save  development  and  acquisition  costs  by  streamlining 
the  acquisition  process  for  low  risk  items  through  Non- 
Developmental  Item  (NDI)  acquisitions. 

b.  To  quickly  field  systems  to  meet  urgent  operational  needs 
accepting  moderate  to  high  risks  through  DA-directed  Limited 
Procurement  (LP)  acquisitions. 

6-2.  Tailoring  T&E 

Each  of  the  above  types  of  acquisition  requires  an  alternative 
T&E  strategy  which  differs  from  the  classic  T&E  strategies  laid 
out  elsewhere  in  this  pamphlet. 


Section  II 

NDI  Acquisition  Process 


6-3 .  NDI  Features 

NDI  acquisitions  provide  a  preferred  alternative  if  the  market 
surveillance  reveals  that  items  are  available  which  have  a  high 
probability  of  meeting  the  users'  requirements. 

a.  NDI  feasibility  may  surface  before  preparation  of  the  MNS 
or  may  be  identified  during  the  market  investigation.  This  is 
based  upon  continuous  market  surveillance,  front-end  analysis, 
responses  to  Mission  Area  Analysis  deficiencies,  and  the  proposed 
solution  in  the  MADP.  The  market  investigation  becomes  much  more 
important  as  a  data  source  for  NDI  systems  and  often  is  the  only 
source  prior  to  a  milestone  I/III  decision  review. 

b.  T&E  requirements  to  support  NDI  acquisition  approaches  do 
not  differ  appreciably  from  the  T&E  requirements  for  a 
developmental  program:  a  TIWG  must  still  be  formed;  a  TEMP  is 
still  required;  test  data  must  be  available;  and  developmental 
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and  operational  evaluations  must  be  performed. 

6-4.  NDI  Tailoring  Opportunities 

NDI  invites  considerable  tailoring  of  the  acquisition  process, 
depending  on  the  extent  of  trade-offs  and  testing  required  to 
verify  achievement  of  system  characteristics  and  technical 
parameters  and  of  operational  effectiveness  and  suitability. 
Maximum  use  should  be  made  of  existing  documentation, 
verification  data,  and  related  evaluations  to  tailor  the 
acquisition.  Documented  results  of  Market  Surveys  or  Market 
Investigations  and  data  from  contractor  testing  may  be  adequate 
to  evaluate  the  system. 

6-5.  Acquisition  of  NDI 

Nondevelopmental  item  acquisition  is  a  generic  term  that  covers 
systems  or  pieces  of  equipment  which  may  require  limited 
development  effort  by  the  Army.  NDI  include  materiel  developed 
and  in  use  by  other  U.S.  military  services  or  government 
agencies,  materiel  developed  and  in  use  by  other  countries,  and 
materiel  available  commercially.  There  are  two  general 
categories  of  NDI  and  a  third  level  of  effort  not  designated  as  a 
separate  category.  Examples  of  this  tailoring  include: 

a.  Basic  NDI.  Basic  NDI  consists  of  off-the-shelf  items 
(e.g.  commercial,  foreign,  other  services)  which  will  be  used  in 
the  same  environment  for  which  they  were  designed  and  will 
require  no  modification.  These  systems  may  be  able  to  undergo  a 
single  decision  review  (Milestone  I/III)  to  verify  sufficiency  of 
the  item  against  the  requirement  and  initiate  type  classification 
with  reduced  MDR  documentation. 

b.  NDI  Adaption.  NDI  Adaption  consists  of  off-the-shelf 
items  to  be  used  in  an  environment  different  than  that  for  which 
designed  and  which  will  require  modification  to  satisfy 
requirements  through  a  highly  abbreviated  EMD  phase.  These 
systems  may  be  able  to  undergo  a  combined  Milestone  I/II  decision 
review  and  a  Milestone  III  decision  review  to  approve  production. 

c.  NDI  Integration.  NDI  Integration  is  focused  on  the 
integration  of  existing  proven  components.  This  category  may 
require  some  hardware  and  software  development  and  integration. 

MS  I  and  MS  II  decisions  occur  very  close  together  and  the 

MS  II  decision  point  may  be  waived  based  on  a  judgement  that 
there  is  no  need  for  development.  NDI  feasibility  surfaces 
during  the  normal  requirements  generation  process  with  the 
preparation  of  an  Mission  Need  Statement  (MNS)  and  a  preliminary 
determination  of  whether  NDI  is  a  viable  option.  This 
determination  by  the  MATDEV  is  based  on  an  initial  analysis  of 
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the  operational  requirements  in  the  MNS  versus  technology  or 
materiel  already  developed  and  in  existence  (e.g.,  foreign-made 
materiel) .  The  criteria  for  a  viable  option  is  that  a  facsimile 
system  or  elements  of  a  system  are  already  operationally 
successful  and  are  adaptable  to  the  operational  requirements 
specified  in  the  MNS. 

6-6.  Advantage  of  NDI 

An  important  advantage  of  NDI  alternatives  is  reduced  acquisition 
time.  This  is  accomplished,  in  part,  by  minimizing  Army  testing 
on  NDI.  General  guidance  is  that  we  will  not  test  when  existing 
date  (contractor  or  other  sources)  affords  an  estimate  of  system 
perfomance  at  a  level  of  confidence  appropriate  to  the  r’iission. 

It  is  imperative  that  independent  evaluators  get  involved  early, 
participate  in  the  formulation  of  the  acquisition  strategy  and 
market  survey  or  investigation  plans,  and  provide  developmental 
or  operational  evaluation  and  assessment  reports. 

6-7.  NDI  Documentation 

The  following  documents  cannot  be  "tailored  out"  of  the 
acquisition  process: 

a.  IPS. 

b.  MNS. 

C.  ORD. 

d.  Acquisition  Strategy. 

e.  COEA. 

f.  Documented  Report  of  Market  Survey  or  Investigation. 

g.  TEMP. 

h.  ILSP. 

i.  SMMP. 

j .  TC  Action. 

6-8.  NDI  Type  Classification  Actions 

Type  classification  (TC)  is  required  for  NDI  acquisitions,  unless 
specifically  exempted  by  regulation. 

a.  TC  Generic.  This  designation  is  used  only  for  NDI  items, 
and  is  the  first  TC  step  when  a  specific  make  and  model  of  an 
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item  are  not  initially  known.  It  is  based  on  performance 
specifications  or  functional  purchase  description,  and  is  used  to 
procure  items  for  T&E.  After  a  manufacturer  is  selected,  all 
test  procedures  and  acceptance  criteria  are  satisfied,  and  the 
make  and  model  number  are  identified,  the  item  must  be  type 
classified  Standard  before  procurement  for  fielding  can  begin. 
This  determination  is  made  by  the  MDA. 

b.  TC  standard.  An  NDI  may  be  type  classified  standard  if  it 
meets  all  provisions  for  TC  Standard.  These  include:  the  item  has 
been  accepted  for  the  operational  mission  intended,  is 
supportable  in  its  intended  environment,  possesses  a  complete 
technical  data  package,  including  ILS  and  maintenance  support. 

c.  Other  interim  TC  designations  may  be  applied  to  NDI, if 
required. 

6-9.  NDI  Acquisition  Process  Flow 

The  acquisition  process  to  be  used  for  an  NDI  strategy  is 
basically  the  same  as  the  established  acquisition  process  for 
developmental  acquisitions.  Process  flow  described  herein  is  a 
"typical”  listing  of  activities  that  would  normally  take  place 
in  an  NDI  acquisition.  Actual  process  activities  may  differ 
somewhat  on  a  case-by-case  basis,  tempered  by  program  specific 
requirements  and  degree  of  tailoring. 

a.  The  NDI  acquisition  process  begins  like  any  other  materiel 
acquisition  with  generation  and  validation  of  a  MNS  prior  to  MS  O 
and  authorization  from  the  MDA  at  MS  0  to  study  alternative 
solutions.  After  milestone  0  approval,  the  MATDEV  and  CBTDEV 
jointly  study  alternative  materiel  concepts  to  identify  the  most 
promising  potential  solution (s)  to  the  validated  user  need. 

b.  NDI  feasibility  surfaces  during  the  normal  process  of 
identifying  alternatives.  After  the  CBTDEV  establishes 
operational  requirements  representing  essential  user  needs,  the 
MATDEV  assesses  technical  feasibility  and  makes  a  preliminary 
determination  of  NDI  suitability. 

c.  The  MATDEV  then  conducts  a  market  survey  or  investigation 
based  upon  the  MNS  and  has  the  objective  of  determining  viability 
of  an  NDI  approach  or  of  the  existence  of  other  streamlining 
opportunities . 

(1)  The  market  survey  or  investigation  is  tailored  to  the 
situation,  and  involves  interaction  between,  and  participation 
by,  the  MATDEV,  user,  independent  evaluators,  testers,  threat 
integrator,  industry,  and  logistician.  The  MATDEV  should  assure 
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that  the  independent  evaluators  review  the  market  survey  or 
investigation  questionnaire  so  that  all  required  data  may  be 
collected. 

(2)  The  MATDEV  should  coordinate  the  requirement  with  the 
International  Materiel  Evaluation  (IME)  Office  at  TECOM  to 
determine  what  is  available  on  the  foreign  market  (See  Section 
IV,  below) . 

(3)  The  CBTDEV  uses  the  results  of  the  effort  to  evaluate 
effectiveness  and  suitability  of  NDI  as  a  potential  solution. 

d.  If  the  market  suggests  potential  NDI  suitability,  the 
MATDEV  and  CBTDEV  conduct  an  iterative  trade-off  process 
evaluating  available  products,  refining  draft  requirements,  and 
determining  acceptable  trade-offs  in  cost,  schedule,  performance, 
and  supportability .  From  this  analysis  the  CBTDEV  prepares  the 
ORD. 

e.  Concurrent  with  initiation  of  a  Market 
Survey/Investigation,  the  MATDEV  establishes  the  TIWG  and 
initiates  preparation  of  the  TEMP.  The  TIWG  plans  and 
coordinates  all  T&E  to  be  conducted  during  the  acquisition 
process  and  assists  in  the  development  of  the  acquisition 
strategy  and  all  supporting  documentation  with  T&E  implications. 
The  TEMP  identifies  critical  operational  issues  and  critical 
technical  parameters  and  outlines  the  approach  that  will  be  used 
to  capture  required  data  to  perform  the  developmental  and 
operational  evaluations.  The  TEMP  also  captures  the  MATDEV 
evaluation. 

f.  The  developmental  evaluator  will  prepare  lEP/IAP  and  the 
operational  evaluator  will  prepare  TEP  as  required  to  document 
specific  data  requirements  and  sources.  These  documents  are 
prepared  in  Phase  0  and  are  applied  in  Phase  I.  The  evaluators 
complete  their  evaluations  and  prepare  an  lER/IAR  or  OA/EOA  that 
are  included  in  the  documentation  at  Milestone  II/III  MDR. 
Although  the  evaluators  are  not  required  to  prepare  plans  and 
reports  to  support  the  market  survey  or  investigation,  the  MATDEV 
should  share  market  data  and  information  with  the  evaluators  and 
solicit  their  input  to  the  conclusions  to  be  presented  at  the 
Milestone  I  MDR. 

g.  The  TIWG  members  determine  the  type  and  amount  of  testing 
required  to  verify  achievement  of  system  characteristics  and 
technical  parameters  and  of  operational  effectiveness  and 
suitability.  The  TIWG  documents  the  planned  testing  in  the  TEMP. 
As  with  all  acquisition  programs,  the  T&E  community  is  encouraged 
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to  make  maximum  use  of  existing  data  and  sources  to  minimize 
testing.  Potential  data  sources  include:  commercial  testing, 
commercial  user  data,  foreign  governments,  foreign  contractors, 
third  party  participants,  and  independent  evaluation  agencies 
such  as  Underwriter  Laboratories  and  Consumer  Reports.  When  data 
are  not  available,  or  when  it  is  suspect,  testing  can  and  should 
be  conducted. 

h.  The  HATDEV  initiates  development  of  an  NDI  acquisition 
strategy,  including  any  recommendation  to  the  MDA  for  tailoring 
out  one  or  more  MDR  from  the  process.  If  the  NDI  solution 
involves  foreign  materiel,  the  Foreign  Comparative  Test  Program 
should  be  considered. 

i.  Based  upon  the  recommendation  of  the  acquisition  team,  the 
decision  to  proceed  with  an  NDI  acquisition  is  made  by  the  MDA  at 
program  initiation,  the  MS  I  decision  point.  The  documentation 
required  for  MDR  vary  by  ACAT  (however,  most  NDI  are  ACAT  III  and 
IV  IPR  programs) .  This  review  determines  the  capability  of  the 
marketplace  to  provide  an  item  for  the  Army  and  approves  the  NDI 
acquisition  strategy. 

j.  Following  approval  of  the  ac(^isition  strategy,  the  MATDEV 
begins  preparation  of  a  system  specification  (preferably 
performance  based)  or  a  functional  purchase  description  for  the 
solicitation.  These  documents  describe  the  military  requirements 
to  industry. 

k.  Before  the  release  of  the  solicitation  document,  the 
Acquisition  Strategy  Report  must  be  approved  by  the  MDA. 

l.  If  required,  PPQT  (government  or  contractor)  will  be 
conducted  before  initiation  of  manufacturing. 

m.  If  required,  new  equipment  training  (NET)  is  conducted. 

NET  may  be  performed  by  the  contractor. 

n.  The  materiel  fielding  plan  is  finalized  and  the  materiel 
fielding  agreement  completed  jointly  by  the  MATDEV  and  the 
gaining  MACOMs.  Materiel  release  will  be  accomplished  in 
accordance  with  AR  700-142,  and  results  in  fielding  of  the  NDI  to 
the  user. 

o.  ILS  is  often  a  particularly  difficult  aspect  of  NDI 
acquisition,  demanding  ardent  management  attention  from  both  the 
MATDEV  and  the  contractor.  A  successful  ILS  effort  can  be 
achieved  only  through  joint  effort  between  the  MATDEV,  CBTDEV  and 
the  contractor.  Emphasis  of  the  early  ILS  effort  is  to  define 
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supportability  related  system  characteristics  to  be  included  in 
the  system  specification,  thereby  influencing  the  ultimate 
selection  of  the  item. 

p.  MANPRINT  is  a  major  consideration  in  determining  whether 
or  not  NDI  is  a  suitable  acquisition  strategy.  MANPRINT 
activities  will  predict  system  demands  on  future  personnel 
inventory,  and  whether  there  are  unsupported  requirements,  i.e., 
human  factors  engineering  issues  and  mental  training  burden. 

Early  NDI  efforts  must  focus  on  identifying  MANPRINT  issues  and 
developing  accommodations  or  "work-arounds."  Where  there  are 
shortfalls,  trade-offs  should  be  conducted  to  determine  the 
impacts  of  modifying  the  NDI  to  correct  shortfalls  versus  the 
consequences  of  accepting  known  MANPRINT  limitations 

r.  Safety,  health,  and  environmental  hazards  should  be 
evaluated  at  MDR  to  determine  whether  they  pose  acceptable  levels 
of  risks  in  accordance  with  system  safety  risk  management 
requirements  of  AR  385-16.  A  comprehensive  system  safety  program 
plan  (MIL-STD-882)  should  be  developed  early  in  the  NDI 
acquisition  program  to  clearly  define  the  system  safety  role  in 
each  phase  of  the  acquisition  process. 

6-10.  RAM  for  NDI 

Quantitative  RAM  requirements  should  be  developed  for  the  NDI. 
Prior  to  the  MS  I  review,  a  tailored  RAM  Rationale  Report  (RRR) 
should  be  prepared  based  on  mission  needs  and  a  thorough  user 
analysis  of  market  surveys  and  investigation  results.  RAM 
parameters  contained  in  the  RRR  will  be  considered  against  the 
characteristics  of  items  available  in  the  marketplace. 

a.  Many  approaches  can  be  taken  to  gather  valid  RAM  data  from 
the  market.  One  approach  is  to  review  any  RAM  analysis  that  the 
manufacturer  performed  in  the  development  of  the  item.  In  market 
surveys  or  investigations,  a  range  of  values  limiting  RAM 
requirements  may  be  used  as  a  baseline  for  the  RAM  assessment. 
When  quantitative  RAM  data  ares  not  available,  it  may  be  possible 
to  assess  relative  RAM  values  or  to  perform  a  qualitative 
assessment  of  RAM  based  on  subjective  feedback  from  existing 
commercial  users. 

b.  If  either  independent  evaluator  determines  that  the  Market 
Survey  or  Investigation  did  not  provide  data  adequate  to  resolve 
RAM  issues,  testing  may  be  required.  The  TIWG  should  be  convened 
to  provide  alternative  solutions  to  satisfy  RAM  issues  for  the 
system.  Evaluators  should  not  demand  RAM  data  in  rigid  formats, 
but  should  be  flexible  in  accepting  and  adapting  available  market 
data  that  can  be  used  to  answer  the  essential  questions. 
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c.  When  Market  Surveys  or  Investigations  or  Army  testing 
demonstrate  that  commercially  available  materiel  cannot  meet  the 
CBTDEV  RAM  requirements,  several  alternatives  exist.  Existing 
commercial  equipment  may  be  modified  to  meet  RAM  requirements,  or 
existing  mission  profiles  may  be  modified  to  see  if  commercially 
demonstrated  RAM  values  are  acceptable.  When  RAM  is  an  extremely 
critical  design  characteristic  and  the  commercial  RAM  parameters 
are  far  inferior  to  the  requirements,  the  NDI  strategy  may  need 
to  be  abandoned  and  a  traditional  development  considered. 


Section  III 
NDI  T&E  Process 


6-11.  Test  and  Product  Assurance  of  NDI 

An  important  advantage  of  NDI  alternatives  is  reduced  acquisition 
time,  partially  due  to  the  ability  to  minimize  Army  testing  on 
NDI.-  Every  effort  should  be  made  to  evaluate  the  achievement  of 
the  system  characteristics  and  technical  parameters  and  of  the 
operational  effectiveness  and  suitability  of  the  NDI  using 
existing  data  from  the  contractor  or  any  other  credible  source. 

If  contractor  and  commercial  user  data  are  not  sufficient,  the 
minimum  amount  of  testing  necessary  to  conduct  he  required 
developmental  and  operational  evaluations  should  be  performed. 

In  all  cases,  however,  a  developmental  lER  or  lAR  and  operational 
TER  or  AOA  will  be  required  to  support  the  MS  III  decision. 

6-12.  Testing  before  Milestone  I 

Testing  prior  to  MS  I  should  be  limited  to  only  that  which  is 
essential  to  support  an  NDI  decision.  These  tests  are  sponsored 
by  the  MATDEV  (usually  TFT  conducted  by  TECOM)  or  the  CBTDEV 
(usually  CEP  tests  conducted  by  OPTEC) ,  not  the  independent 
evaluators.  These  tests  are  an  extension  of  the  market 
survey/investigation  effort.  Prior  to  any  dedicated  Army 
testing,  the  following  sources  should  be  searched  for  relevant 
data: 

a.  Manufacturer's  test  results. 

b.  Manufacturer's  specifications  or  technical  data  packages. 

c.  User  failure  data. 

d.  Independent  testing  agencies  test  results, 

6-13.  Testing  after  Milestone  I 

The  type  and  amount  of  testing  that  will  be  performed  after  MS  I 
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will  be  determined  by  the  TIWG  members  and  documented  in  the 
TEMP.  Testing  and  independent  evaluations  will  be  in  accordance 
with  the  lEP/IAP  and  TEP.  The  TIWG  members  should  minimize 
testing  needs  as  much  as  possible  and  maximize  the  use  of 
existing  data  to  perform  the  evaluations  and  assessments. 

a.  DT  and  OT.  DT  and  OT  should  be  limited  to  data 
acquisition  that  is  essential  to  the  decision  making  process,  and 
for  which  there  are  no  existing  data  available.  When  both  DT  and 
OT  are  required,  maximum  effort  should  be  made  to  combine  the 
testing. 

b.  Developmental  and  Operational  Evaluation.  NDI  acquisitions 
require  evaluations  by  both  the  DIE  and  lOE.  lER/IAR  or  TER/AOA 
are  provided  at  each  MDR  by  the  evaluators.  Every  effort  should 
be  made  to  perform  these  evaluations  using  existing  data. 

6-14.  Post  MS  III  Testing 

Testing  of  the  NDI  after  MS  III,  if  required,  is  oriented  to 
qualification  of  the  manufacturing  process  and  compliance  with 
the  technical  data  package,  validation  and  refinement  of 
operating  and  support  cost  data,  RAM  characteristics,  logistics 
support,  training,  and  provisioning. 

6-15.  Test  and  Evaluation  Requirements  for  NDI 
No  acquisition,  including  NDI,  is  exempt  from  DTE  and  OTE 
necessary  to  verify  MANPRINT,  quality,  safety,  RAM,  performance, 
logistics  supportability  and  transportability  characteristics  of 
a  system.  Tests  by  manufacturers  and  contractors,  previous 
performance  data,  and  market  analysis  information  may  validate 
acceptability  of  these  characteristics  and  provide  evidence  of 
system  operational  effectiveness  and  suitability.  However,  for 
the  production  decision  at  least  minimal  T&E  is  required.  OT&E 
of  NDI  systems  must  follow  standard  DOD  policy  with  regard  to 
system  contractor  involvement. 

6-16.  Testing  Prior  to  Initial  MDR 

Testing  should  be  limited  to  that  absolutely  necessary  to  obtain 
data  necessary  to  make  the  NDI  decision.  This  is  accomplished 
through  the  Market  Survey  or  Investigation. 

a.  As  an  initial  step,  the  Army  will  minimize  testing  by; 

(1)  Obtaining  and  assessing  contractor  test  results. 

(2)  Obtaining  usage/failure  data  from  other  customers. 

(3)  Obsejrving  contractor  testing. 
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(4)  Obtaining  test  results  from  independent  test 
organizations,  e.g..  Underwriter's  Laboratory,  National  Bureau  of 
Standards . 


b.  If,  based  on  this  initial  data  collection,  it  is  decided 
that  more  information  is  needed  to  make  a  sound  NDI  decision,  the 
MI  may  enter  into  an  evaluation  phase.  NDI  candidates  may  be 
bought  or  leased,  and  DT  (TFT)  and/or  OT  (including  RAM  and 
logistic  support)  should  be  conducted  in  the  operational/combat 
environment.  Safety  release  procedures,  in  accordance  with  AR 
385-16,  must  be  followed  prior  to  conducting  OT.  The  results 
will ; 


(1)  Directly  support  the  AS  to  accept  or  reject  the  NDI 
alternative. 

(2)  Influence  preparation  of  requirements  documents. 

(3)  Assist  in  preparation  of  solicitation  documents. 

c.  The  test  results  will  not  be  used  to  select  a  specific 
contractor  or  product. 

6-17.  Testing  After  the  Initial  MDR  ^  ^ 

All  testing  after  the  initial  MDR  must  be  described  and  justified 
in  the  TEMP  and  specifically  approved  by  the  program  decision 
authority . 

6-18.  DTE  for  NDI  ^  ... 

No  DT  is  conducted  unless  the  DIE  identifies  specific  information 
needs  that  cannot  be  satisfied  by  contractor  or  other  test  data 
sources.  The  DIE  will  not  demand  data  in  rigid  formats,  but  will 
be  flexible  in  accepting  and  adapting  data  that  answer  essential 
questions. 

a.  Risks  associated  with  hardware/software  modifications  (NDI 
category  B)  and  the  third  level  of  NDI  effort  involving 
integration  of  components  will  be  carefully  considered  when 
determining  test  requirements.  DT  requirements  are  tailored  to 
each  specific  system. 

b.  Conduct  (as  a  minimum)  DTE  to  verify  integration  and 
interoperability  with  other  system  elements.  Additional  T&E,  as 
appropriate,  will  be  conducted  to  evaluate  and  control  risk. 

PPQT  and  PQT  should  be  identically  designed.  If  the  PPQT  is 
completely  successful,  the  PQT  can  be  waived  in  favor  of  a  FAT. 

If  the  PPQT  is  partially  successful,  the  PQT  can  be  redesigned  to 
address  only  those  parameters  which  are  still  in  question. 
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c.  The  following  is  provided  as  general  guidance,  not  rigid 
requirements,  of  the  testing  activities  appropriate  for  the 
following  NDI  options: 

(1)  Off-the-shelf  items  to  be  used  in  the  same  environment 
for  which  they  were  designed  (i.e.,  no  development  or 
modification  of  hardware  or  software  is  required)  will  normally 
not  require  developmental  testing  before  MS  I/II  or  I/III? 
however,  TFT  may  be  conducted  to  support  the  MS  decision.  When 
the  contract  is  awarded  to  a  contractor  who  has  not  previously 
produced  an  acceptable  finished  product  and  the  item  is  assessed 
as  high  risk,  a  PQT  should  be  required. 

(2)  Those  off-the-shelf  items  which  require  some 
modification  of  hardware  or  operational  software  (e.g., 
militarization  or  rugged iz at ion)  will  require  TFT,  unless  the 
decision  authority  documents  that  further  testing  is  not 
required.  PPQT  is  required  if  feasibility  testing  results  in  the 
necessity  for  fixes  to  the  item.  PQT  is  required  to  support 
materiel  release. 

(3)  An  R&D  effort  is  required  for  NDI  items  used  as 
subsystems,  modules,  or  components  which  contribute  to  a  materiel 
solution.  Systems  engineering,  software  modification,  and 
testing  is  required  to  ensure  a  total  system  meets  user 
requirements  and  is  producible  as  a  system.  TFT  is  required  in  a 
military  environment.  PPQT  of  the  complete  system  is  required. 
Hardware/computer  software  integration  tests  are  required  and  PQT 
is  required. 

d.  Some  follow-on  testing  of  the  NDI  may  be  required  to 
verify  the  adequacy  of  corrective  actions  indicated  by  the  PQT. 

6-19.  OTE  for  NDI 

NDI  does  not  automatically  mean  no  OT  will  be  required.  If  the 
MATDEV  demonstrates  through  Market  Survey  or  Investigation  data 
that  NDI  products  will  satisfy  the  requirements  document,  OT  may 
not  be  required  if  the  lOE  concurs.  This  determination  must  be 
included  in  the  initial  MDR  documentation  (IPS  and  TEMP)  and 
approved  by  the  MDA. 

a.  For  Basic  NDI,  lOT  will  not  normally  be  required. 

b.  For  NDI  Adaptation,  lOT  will  be  required  only  when 
critical  issues  in  the  TEP  have  not  been  addressed.  Prior 
concurrence  by  the  lOE  is  required  to  eliminate  lOT. 

c.  For  NDI  integration,  lOT  is  always  required. 
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d.  Follow-On  Evaluation.  Testing  of  the  NDI  after  the  first 
unit  is  equipped  is  oriented  to  the  validation  and  refinement  of 
operating  and  support  cost/data,  RAM  characteristics,  logistic 
support,  training,  provisioning  planning,  etc.  These  tests  can 
materially  aid  the  logisticians  in  supporting  NDI  throughout  its 
life-cycle. 

6-20.  Test, Requirements  by  Type  of  NDI 

Testing  requirements  will  be  tailored  to  each  specific  system. 

The  following  test  guidance  by  NDI  category  is  not  a  rigid 
requirement,  but  general  characteristics  of  testing  activities 
appropriate  to  each  NDI  category.  The  goal  of  minimum  testing 
still  remains  regardless  of  NDI  category. 

a.  Basic  NDI.  No  testing  prior  to  PQT  except  when  the 
contract  awarded  to  a  contractor  who  has  not  previously  produced 
acceptable  finished  products  and  the  item  is  assessed  as  high 
risk.  In  that  case,  PPQT  should  be  required. 

b.  NDI  Adaptation.  Feasibility  testing  is  required  in  the 
military  environment.  PPQT  is  required  if  feasibility  testing 
results  in  fixes  to  the  item.  PQT  is  required.  Limited  user 
evaluation  may  occur  during  feasibility  and/or  preproduction 
tests. 

c.  Third  Level  of  NDI  Effort  (Integration  of  Components) . 
Feasibility  testing  is  required  in  military  environment.  PPQT  of 
complete  system  is  required.  Hardware  and  computer  software 
integration  tests  required.  lOT  is  required.  PQT  is  required. 


Section  IV 

Foreign  Comparative  Testing  (FCT)  Program 


6-21.  FCT  Mission 

The  mission  of  the  FCT  program  is  to  provide  cost  effective 
foreign  equipment  alternatives  that  meet  approved  Army 
requirements,  and  which,  after  being  successfully  tested  and 
evaluated,  can  be  selected  in  a  procurement  decision.  The  FCT 
involves  T&E  of  weapon  systems,  equipments,  and  technologies  of 
allied  and  other  friendly  nations  with  a  view  toward  meeting 
valid  existing  Army  requirements  while  reducing  duplication  in 
R&D,  enhancing  standardization  and  interoperability,  improving 
cooperative  support,  and  promoting  competition  and  international 
technology  exchange. 

6-22.  FCT  Procedures 
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The  FCT  program  generally  fits  into  the  Army  acquisition  cycle  as 
part  of  the  normal  T&E  process  of  NDI  materiel.  FCT  is  not  a 
short  cut  to  fielding,  but  can  achieve  significant  savings  in 
time  and  funding  versus  traditional  development  as  R&D  is  not 
required.  Procedures  and  criteria  for  project  submissions  are 
contained  in  DoD  5134. M-2.  The  following  general  procedures 
apply  for  Army  FCT  implementation: 

a.  A  MATDEV,  acting  as  the  project  proponent,  can  sponsor  an 
item  by  preparing  a  Candidate  Nomination  Proposal  (CNP)  for  the 
Army  FCP  Executive  Agent,  currently  AMC.  After  verifying  that  the 
DoD  FCT  criteria  have  been  met  and  coordinating  the  CNP  with 
appropriate  Army  organizations,  the  CNP  will  be  forwarded  to  DoD 
through  SARD-DI  for  funding.  Informal  coordination  of  draft  CNP 
and  joint  working  groups  on  proposed  FCT  projects  is  encouraged. 

b.  Upon  approval  of  the  CNP,  detailed  plans  for  developmental 
and  operational  evaluations  will  be  prepared  by  the  DIE  and  lOE 
and  coordinated  with  the  acquisition  community.  Foreign  and 
contractor  data  will  be  used  to  the  maximum  extent  possible  to 
satisfy  evaluation  requirements.  If  sufficient  data  are  not 
available,  test  items  will  be  obtained  from  the  foreign  country 
by  way  of  loan,  lease,  or  purchase;  whichever  is  most 
advantageous  to  the  Army  and  agreed  to  by  the  foreign  country. 

c.  DoD  will  provide  FCT  funds  directly  to  the  Army  FCT 
Executive  Agent  who  will  distribute  funding  to  MATDEV  as 
required/approved.  All  required  plans  and  reports  will  flow 
through  the  Army  FCT  Executive  Agent  which  will  provide  Army 
policy  and  oversight  of  all  FCT  projects. 


Section  V 

T&E  of  Limited  Procurement  (LP)  Systems 


6""23  LP  Process 

LP  type  classification  (formerly  called  Limited  Procurement- 
Urgent  or  LP-U)  is  used  when  a  materiel  item  is  required  for 
special  use  for  a  limited  time.  The  specified  limited  quantity 
for  the  LP  item  will  be  procured  without  intent  of  additional 
procurement  of  the  item  under  this  classification.  The  LP  type 
classification  is  used  to  meet  URGENT  operational  requirements 
that  cannot  be  satisfied  by  an  item  type  classified  Standard 
(STD)  . 

6-24.  LP  Criteria 

Criteria  for  LP  type  classification  of  an  item  required  for 
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urgent  operational  use  will  include  the  following. 

a.  Existence  of  an  urgent  operational  requirement, 
substantiated  by  the  using  command  representative  and  CBTDEV  or 
by  HQDA. 

b.  Determination  that  there  is  no  type  classified  item  that 
fully  satisfies  the  requirement. 

c.  Sufficient  definition  of  the  military  characteristics  of 
the  item  in  materiel  requirements  documents  to  allow  subsequent 
evaluation  of  the  item. 

d.  Demonstration  that  the  proposed  item  does  not  qualify  for 
STD  and  offers  no  more  than  a  moderate  risk. 

e.  Determination  that  the  proposed  item  can  be  economically 
maintained  and  logistical ly  supported  in  the  geographic  area  and 
time  frame  for  which  the  type  classification  is  valid. 

6-25.  Prohibitions  Against  Misuse  of  LP  type  classification 
Type  classification  LP  will  not  be  used  solely  to  avoid  the 
checks  and  balances  of  the  acquisition  process  or  to  avoid  T&E  of 
the  item. 

6-26.  Operational  Field  Evaluations 

Not  later  than  six  months  following  delivery  of  the  initial 
shipment  of  the  LP  item,  the  user  or  requester  of  the  item  will 
collect  data  and  provide  an  operational  field  evaluation 
statement  to  the  program  manager  or  mission  assignee  agency. 
Information  copies  will  be  provided  to  HQDA  (SARD-RPP) ,  TRADOC, 
AMSAA,  OPTEC,  and  MRS A. 

6-27.  Expedited  OTE  for  LP  Systems 

As  the  independent  operational  evaluator  and  the  primary  user 
tester  for  the  Army,  OPTEC  can  perform  expeditious  operational 
assessments  and  supporting  LUT  to  support  LP  procurement  prior  to 
materiel  release  to  the  first  unit  equipped  if  the  urgent 
requirement  permits.  OPTEC  participation  in  LPU  procurement  can 
cover  a  spectrum  of  involvement  as  follows  (This  methodology  is 
applicable  to  both  war  and  non-wartime  urgent  procurement) . 

a.  Participation  in  a  materiel  release  decision  by  rendering 
an  abbreviated  operational  assessment  (AOA)  of  the  system  based 
on  program  documentation  and  contractor/developmental  testing. 

b.  Participation  in  a  materiel  release  decision  by  rendering 
an  AOA  of  the  system  based  on  program  documentation  and  a 
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combined  DT/OT  run  by  TECOM. 

c.  Participation  in  a  materiel  release  by  rendering  a  TER 
based  on  results  of  a  quick  reaction  LUT  in  addition  to  results 
of  contractor  testing  or  DT. 

6-28.  Quick  Reaction  LUT 

The  procedures  for  the  conduct  of  a  quick  reaction  LUT  are  as 
follows: 

a.  Upon  receipt  of  requirements  for  a  quick  reaction  LUT,  the 
evaluator  and  the  tester  will  jointly  prepare  a  TEP  using  CBTDEV 
COIC,  if  available.  The  evaluator  will  develop  additional  issues 
for  the  TEP.  If  there  are  no  COIC,  the  evaluator  developed 
issues  will  be  considered  as  the  critical  issues. 

b.  Evaluator  issues  will  include  the  following  issues  in 
order  of  priority,  but  are  not  limited  to  these  issues: 

(1)  Does  the  system  have  reliability  or  performance 
deficiencies  which  endanger  survivability  of  unit  or  personnel? 

(2)  Does  the  system  have  reliability  or  performance 
deficiencies  which  endanger  performance  of  the  mission? 

(3)  Does  the  system  have  critical  or  catastrophic  safety 
deficiencies? 

(4)  Does  the  system  have  reliability,  maintainability  or 
performance  deficiencies  which  cause  excessive  logistics  burden 
to  using  unit  or  supporting  units? 

(5)  Does  the  system  have  reliability,  maintainability  or 
performance  deficiencies  which  cause  excessive  life  cycle  costs? 

(6)  What  added  margin  of  capability  does  the  system 
provide  to  the  using  unit  or  the  force? 

c.  The  evaluator  and  tester  will  prepare  a  Pattern  of 
Analysis  in  conference,  developing  OTMOP  to  address  the  issues 
(see  Part  Five  for  details  of  these  OTE  peculiar  procedures) . 
These  OTKOP  will  form  the  basis  of  the  test  design.  The  LUT 
will,  as  the  name  implies,  be  a  limited  test  in  scope  and 
duration.  The  LUT  will  not  be  comparable  to  an  lOTE  and  will  not 
be  so  structured.  Given  the  joint  development  of  the  Pattern  of 
Analysis  and  the  limited  scope  of  test,  the  evaluator  and  the 
tester  should  be  able  to  complete  the  TEP  in  fifteen  days  or 
less. 
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d.  Upon  approval  of  the  TEP,  an  OTRR  will  be  held.  If  all 
test  assets  are  available,  test  player  training  will  begin 
followed  by  testing  for  record.  Necessary  test  assets  include 
test  items,  ammunition,  support  items  of  equipment,  the  system 
support  package  and  all  other  test  support  packages,  test  funds, 
test  ranges  and  maneuver  areas,  test  player  personnel,  and  test 
directorate  personnel  all  of  which  must  be  present  to  begin 
training  of  player  personnel  and  start  of  test. 

e.  Upon  completion  of  test,  a  TER  will  be  jointly  prepared  by 
the  evaluator  and  tester.  The  TER  will  have  strict  page  limits 
and  will  be  approved  and  available  to  decision  makers  within 
thirty  days  after  completion  of  the  test.  The  evaluator  will 
also  prepare  an  executive  summary  type  briefing  of  the  TER. 


Section  VI 

Accelerated  Software  Development  Process  for  Information  Systems 


6-29.  General 

Technological  changes  have  occurred  that  allow  software 
development  processes  that  are  different  from  the  traditional 
approaches  (i.e..  Grand  Design,  Waterfall).  Software  development 
has  been  enhanced  by  the  availability  of  automated  tools  that 
help  define  requirements;  help  design  and  document  the  system; 
generate  code;  help  with  configuration  management;  and  make 
maintenance  easier  by  developing  embedded  test  instrumentation. 
These  procedures  can  allow  for  faster  production  of  information 
systems  at  less  cost. 

a.  Features  of  an  accelerated  software  development  process 
(ASDP)  include  incremental  blocks  of  development  and  testing;  use 
of  a  high-level  functional  description  (HLFD)  or  ORD;  user 
involvement  throughout  the  prototyping  process;  MS  III.C 
(Certification)  of  an  operationally  tested  representative  sample; 
MS  III.F  (Final)  which  authorizes  fielding  of  the  final  block; 
and  milestone  decision  reviews  delegations  to  Project  Boards. 

b.  The  procedures  outlined  in  this  section  offer  an 
alternative  to  the  standard  process  described  in  Chapter  5  for 
small  to  medium  information  systems  (i.e.,  classes  III  and 
lower) . 

6-30.  Accelerated  Software  Development  Strategy 

This  paragraph  briefly  outlines  the  life  cycle  management  model 
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for  the  ASDP,  and  discusses  the  T&E  activities  related  to  each 
phase  of  the  model.  A  comparison  of  this  strategy  with  the 
standard  strategy  described  in  Chapter  5  can  be  helpful  in 
understanding  the  mechanism  of  the  ASDP. 

a.  Determination  of  the  Mission  Need. 

(1)  Acguisition  activities.  Activities  to  be  completed 
prior  to  Milestone  0  are  outlined  below.  These  actions  will 
culminate  in  a  defined  mission  needi  and  produce  the  MNS. 

(a)  Identification  of  a  need  by  means  of  completion  of 
an  Information  Requirements  Study,  modeling  of  the  business 
processes,  or  identifying  requirements  through  the  operation  of 
existing  systems  or  processes. 

(b)  Evaluation  of  the  identified  need  is  evaluated  to 
determine  if  it  can  be  satisfied  by  a  nondevelopmental  solution, 
such  as  changes  in  doctrine,  operational  concepts,  training,  or 
organization. 

(c)  An  initial  evaluation  of  the  resources  required  to 
develop  a  solution.  The  preferred  method  would  be  through  a 
Functional  Economic  Analysis  (FEA) .  The  FEA  does  not  replace  the 
Economic  Analysis  (EA)  required  after  Milestone  0. 

(2)  T&E  and  CE  Activities.  Typically  no  T&E  or  CE 
activities  are  conducted  in  this  phase. 

(3)  Milestone  0  (Concept  Studies  Decision).  Approval  of 
the  MNS  is  required  by  this  milestone. 

b.  Concept  Exploration  and  Definition. 

(1)  Acquisition  activities.  Activities  to  be  completed 
prior  to  Milestone  I  are  outlined  below.  These  actions  will 
culminate  in  a  coordinated  strategy  to  satisfy  the  mission  need 
and  will  produce  the  HLFD. 

(a)  Evaluate  alternative  technical  concepts  and  analyze 
the  technical  risks  to  ensure  that  alternative  system  design 
concepts  adequately  reflect  a  broad  segment  of  the  technology 
base  and  provide  an  acceptable  competitive  environment. 

(b)  Development  of  an  initial  EA. 

(c)  Development  of  the  AS. 
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(2)  T&E  and  CE  Activities.  Typically  no  developmental  or 
operational  testing  is  conducted.  The  TIWG  is  established  in 
this  phase  by  the  PM  (See  Chapter  5) .  CE  activities  include 
participation  in  the  development  of  the  HLFD,  the  preliminary 
TEMP,  and  associated  documents  such  as  training  plans,  the  SMMP, 
and  the  ILSP. 

(3)  Milestone  I  (Concept  Demonstration  Decision) .  T&E- 
related  requirements  for  this  milestone  include  approval  of  the 
HLFD,  the  initial  COIC,  the  CMFs,  and  the  preliminary  TEMP. 

c.  Demonstration  and  Validation. 


(1)  Acquisition  activities.  Activities  to  be  completed 
prior  to  Milestone  I  are  outlined  below.  These  actions  will 
culminate  in  a  demonstration  that  better  defines  the  critical 
design  characteristics  and  expected  capabilities,  that  proves 
that  the  critical  technologies  can  be  incorporated  into  the 
system,  that  the  processes  are  understood  and  attainable,  and 
that  the  first  incremental  block  is' functional  and  ready  for 
final  development  and  testing. 

(a)  Selection  and,  if  necessary,  acquisition  of  the 
developmental  tool  set. 

(b)  Prototyping  the  system  to  conform  with  the  HLFD. 

(c)  Update  the  system  design  based  on  the  prototyping, 
to  include  trade-offs  between  software,  hardware,  firmware,  and 
human  factors. 


(d) 

involvement. 


Using  the  developmental  tools  and  the  user's 
design  block  1. 


(e)  Update  the  AS. 


(f)  Establish  a  Developmental  Baseline. 


(g)  Completion  of  the  EA  for  Block  1,  to  include 
estimates  for  the  other  blocks. 


(h)  Determination  of  the  membership,  and  drafting  of  the 
charter  for  the  Project  Board. 

(2)  T&E  and  CE  Activities.  Typically  no  developmental  or 
operational  testing  is  conducted  prior  to  MS  II.  TIWG  meetings 
are  held  as  required.  CE  activities  include  participation  in 
updating  the  HLFD  and  transitioning  it  into  the  FD  as  the 
development  continues,  updating  the  preliminary  TEMP,  and 
updating  the  associated  documents  such  as  training  plans,  the 
SMMP,  and  the  ILSP. 
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(3)  Milestone  II  (Development  Decision).  MS  II  approves 
the  detailed  design  of  Block  1  and  authorizes  both  the  completion 
of  Block  1  and  the  start  of  the  development  of  subsequent  blocks 
as  resources  become  available.  Approval  of  the  FD,  the  COIC 
update,  and  the  TEMP  update  are  required  by  this  milestone. 

d .  Deve 1 opment . 

(1)  Acquisition  Activities.  Project  Board  reviews  will 
occur  at  key  points  in  the  development  of  the  blocks  ^  of  the 
system  to  ensure  the  project  is  on  track.  These  reviews  have 
been  labeled  "MS  II. X"  to  keep  the  terminology  consistent.  The 
reviews  are  not  MAISRC-level  reviews.  As  developmental  resources 
are  made  available,  the  remaining  blocks  are  prototyped, 
designed,  developed,  integrated  with  previous  blocks,  and  tested. 
Actions  to  be  completed  by  Milestone  III.C  are  outlined  below. 
These  actions  will  culminate  in  the  fielding  of  the 
representative  sample  of  the  system  to  the  Army. 

(a)  Milestone  II. Oi  A  LUT  is  conducted  after  the  MS  II 
review  by  the  independent  operational  tester.  The  LUT  is 
designed  to  test  the  target  hardware.  Commercial  Off— the— Shelf 
(COTS)  software,  and  communications  without  any  application 
software.  This  test  shall  be  be  conducted  before  Block  1  is 
tested  on  the  target  system.  The  Project  Board  conducts  the  MS 
II. 0  review. 

(b)  Milestone  II. 1.  After  MS  II. 0,  Block  1  is  tested  on 
the  target  hardware  to  support  MS  II. 1.  Testing  should  include  a 
software  qualification  test  and  a  LUT.  If  Block  1  is  not  the 
representative  sample.  Milestone  II. 1  will  authorize  the  fielding 
of  Block  1  to  the  operational  testbed.  The  Project  Board 
conducts  the  MS  II. 1  review. 

(c)  Milestone  II. 2  through  II. N.  Each  block  prior  to 
reaching  a  representative  sample  will  be  tested  as  in  (b)  above 
and  reviewed  by  the  Project  Board  with  a  MS  II. X  review.  The 
software  qualification  test  and  LUT  will  examine  the 
functionality  of  the  block  and  their  integration  with  previously 
built  blocks.  This  review  will  authorize  the  fielding  of  the 
block  to  the  testbed.  The  Project  Board  conducts  the  MS  II. X 
reviews . 

(d)  Milestone  III.C  (Certification) .  When  the 
cumulation  of  the  integration  of  the  blocks  comprises  a 
representative  sample  which  has  tested  and  evaluated  using  the  MS 
II. X  approach,  the  system  is  ready  for  a  MS  III.C.  MS  III.C  is 
the  decision  point  that  certifies  the  completed  increments  for 
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fielding  Army  wide.  It  determines  whether  the  completed 
representative  sample  satisfies  the  mission  and  is  ready  for 
deployment.  MS  III.C  requires  a  MAISRC  review.  Approval  by  the 
MAISRC  at  MS  III.C  authorizes  the  expenditure  of  resources  for 
the  deployment  of  the  representative  sample  and  the 
hardware/communications  packages  Army  wide. 

(2)  T&E  hnd  CE  Activities.  The  independent  operational 
evaluator  will  conduct  an  lOT  of  the  representative  sample  to 
support  MS  III.C.  Extensive  use  of  simulation  and  emulation  may 
be  required  to  fully  stress  the  target  configuration.  The  object 
of  the  "fully  stress"  requirement  is  to  ensure  that,  as 
additional  blocks  are  added  beyond  the  representative  sample,  the 
system  will  continue  to  function  without  adverse  impacts  on  the 
user  and  without  the  need  for  expensive  hardware  upgrades.  Test 
plans,  test  reports,  evaluations  and  assessments  will  be  prepared 
by  the  independent  developmental  and  operational  testers  and 
evaluators  (See  Chapter  5)  to  support  the  T&E  during  development. 
The  evaluations  and  assessments  will  be  provided  to  the  Project 
Board  and  MAISRC  as  required.  CE  activities  also  include 
participation  in  the  TEMP  update  for  MS  III.C  and  associated 
documents  such  as  training  plans,  the  SMMP,  and  the  ILSP. 

e.  Production  and  Deployment. 

(1)  Acquisition  Activities.  This  phase  begins  with  MS 
III.C  and  ends  with  MS  III.F  (Final).  MS  III.F  will  approve 
fielding  the  final  block  of  the  system.  The  blocks  completed 
after  MS  III.C  will  be  reviewed  by  the  Project  Board  prior  to 
fielding.  The  reviews  associated  with  these  blocks  will  be 
designated  as  MS  III.l  through  III.M  until  the  final  block  is 
ready  for  fielding.  Because  the  representative  sample  has  been 
fielded  Army  wide,  MS  III.l  would  authorize  Army  wide  fielding  of 
the  first  block  completed  after  MS  III.C. 

(2)  T&E  and  CE  Activities.  These  activities  are  similiar 
to  those  discussed  in  the  MS.II.X  sequence.  The  independent 
operational  evaluator  will  conduct  an  lOT  of  the  final  block  of 
the  system  to  support  MS  III.F.  CE  activities  also  include 
participation  in  the  TEMP  update  for  MS  III.C  and  associated 
documents  such  as  training  plans,  the  SMMP,  and  the  ILSP. 

(3)  Milestone  III.F  (Final).  MS  III.F  is  the  MAISRC  review 
that  determines  that  the  final  block  is  complete,  the  total 
system  is  complete,  the  system  satisfies  the  mission  need,  and 
the  system  is  operationally  effective  and  suitable.  This 
milestone  marks  the  transition  of  the  system  to  operations  and 
support . 
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f.  Operations  and  Support.  Following  MS  III.C,  the  fielded 
blocks  are  in  the  operations  and  support  phase.  The  entire 
system  transitions  to  operations  and  support  after  MS  III. F. 

The  acquisition  process,  T&E,  and  CE  activities 

forward  are  similar  to  the  standard  process  (See  Chapter  5) . 
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Chapter  7 

Test  and  Evaluation  in  Support  of  Systems  Modernization 


7-1.  General 

a.  In  1990,  the  Army  changed  the  process  for  managing 
modifications  to  developed  materiels.  Previously,  Product^ 
Improvement  Proposals  (PIPs)  on  fielded  materiel  were  submitted 
by  the  system  manager  to  HQDA  for  approval,  while  Engineering 
Change  Proposals  (ECPs)  were  initiated  and  approved  by  field 
level  engineering  organizations.  The  result  was,  in  many  cases, 
significant  scrutiny  of  routine  PIPs  on  one  hand,  and  minimal 
scrutiny  to  ECPs  that  had  dramatic  impact  on  costs  and  field 
usage  on  the  other.  New  procedures  were  implemented  to  bring 
both  ECPs  and  PIPs  under  one  process. 

b.  Modification  encompasses  all  hardware,  firmware,  and 
software  changes  except  Class  II  ECPs,  to  materiel  Type 
Classified  Standard,  LRP,  and  LPU,  and  provides  a  single 
comprehsnsive  process  to  manage  modifications.  Complete  details 
on  the  Modification  process  are  contained  in  the  "Interim 
Operating  Instructions  for  US  Army  Materiel  Change  Management", 
dated  6  September  1990. 

c.  All  system  modifications  must  undergo  evaluation,  and  most 
will  require  testing  also.  T&E  is  conducted  to  assure  that  the 
modification  achieves  the  desired  effect  without  degrading 
performance,  reliability,  safety,  or  system  logistical 
characteristics . 

d.  In  order  to  allow  a  reduced  level  of  T&E  effort  on 
modifications  that  do  not  effect  system  performance  or  the 

user  in  any  substantive  way,  an  abbreviated  T&E  process  has  been 
developed  that  eliminates  the  need  for  TIWGs,  TEMPs  and 
Independent  Evaluations.  The  intent  is  to  provide  a  level  of 
T&E  effort  commensurate  with  risk,  recognizing  that  some 
modifications  represent  minimal  risk.  Most  modification  efforts 
will  require  compliance  with  the  "conventional"  T&E  process  as 
described  throughout  this  Pamphlet;  however,  if  the  criteria 
described  in  this  chapter  are  met,  the  abbreviated  process  can  be 
utilized. 

e.  The  term  "program  sponsor"  is  used  to  indicate  the  actual 
manager  of  the  modification  effort.  The  program  sponsor  could  be 
a  PM  or  a  Project  Officer  for  systems  that  have  not  transitioned 
into  production,  or  the  item  manager  for  developed  systems. 
Modifications  to  systems  that  have  transitioned  into  production 
but  that  require  a  significant  effort  may  be  assigned  to  a 


7-1 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


PM/MATDEV  for  development/ implementation.  In  these  cases,  the 
initial  activities  of  the  Modification  effort  will  be  conducted 
by  the  item  manager  until  assignment  of  a  PM/MATDEV. 

7-2.  System  Modification  Requirements 

a.  The  modification  process  currently  in  effect  within  the 
Army  has  four  "program”  levels  as  follows: 

(1)  Modifications  with  the  Configuration  Control  Board 
(CCB)  or  Configuration  Manager  (CM)  as  the  decision  authority. 

(2)  Modifications  with  the  program  sponsor  as  the  decision 
authority. 

(3)  Modifications  with  PEOs/MSCs  as  decision  authority. 

(4)  Modifications  with  the  AAE  as  decision  authority. 

b.  All  modifications  with  the  CCB/CM  as  decision  authority 
can  use  the  abbreviated  T&E  process  as  described  below.  All 
Modifications  with  the  PEO/MSC  or  AAE  as  the  decision  authority 
must  conduct  a  conventional  T&E  program  as  outlined  in  this 
Pamphlet.  Modifications  with  the  program  sponsor  as  the  decision 
authority  must  determine  if  the  abbreviated  process  can  be  used 
as  follows: 


(1)  Total  estimated  budgetary  cost  of  less  than  $5M. 

(2)  No  retrofit  by  MWO  is  required. 

(3)  No  change  in  operational  capability /No  critical 
Operational  Issues  &  criteria  (COIC) . 

(4)  No  change  in  program  cost/schedule/performance 
baseline. 


(5)  No  high  level  interest  (i.e.,  DOD,  congress). 

(6)  No  safety  related  change  to  system. 

(7)  Not  a  Class  I  software  change  lAW  DOD  STD-2167. 


c.  Exceeding  any  of  the  criteria  listed  in  the  abbreviated 
TEMP  will  require  a  Test  Integration  and  Working  Group  (TIWG)  to 
be  convened.  The  TIWG  will  determine  details  and  amount  of 
developmental  and  operational  testing  required  for  the 
modification.  The  TIWG  members  will  tailor  the  TEMP  to  the 
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specific  modification  under  consideration.  Specific  guidance  for 
the  TEMP  format  is  contained  in  DOD  5000. 2M,  Part  7. 

d.  It  is  essential  that  the  MATDEV  involve  the  T&E  community 
early  on  in  the  modification  process.  The  results  of  the  T&E 
will  be  used  by  the  independent  developmental  evaluator, 
operational  evaluator  and  logistician  to  render  assessments 
supporting  or  opposing  the  production  decision  of  the 
modification.  Additionally,  the  evaluators  will  use  the  T&E 
information  gathered  to  render  opinions  on  materiel  release  of 
the  modification. 

e.  Testing  consolidation.  To  achieve  maximum  T&E 
efficiencies,  testing  of  multiple  modifications  into  a  single 
system/end  item  is  encouraged.  This  process  should  be  planned 
thoroughly  early  in  the  modification  process.  One  comprehensive 
TEMP  should  integrate  as  many  minor  changes  as  possible. 

(1)  The  program  sponsor  completes  the  "Combat  Developer 
Coordination  Checksheet"  listed  at  Figure  7-1,  and  forwards  the 
completed  checksheet  along  with  the  Modification  data  package  to 
the  Combat  Developer  for  review. 

(2)  The  combat  developer,  upon  completion  of  this  review, 
must  formally  notify  the  program  sponsor  whether  Critical 
Operational  Issues  (COI)  associated  with  the  modification  have 
been  identified. 

(3)  If  the  Combat  developer  does  not  identify  any  COI 
associated  with  the  Modification,  the  program  sponsor  can  use  the 
abbreviated  T&E  process  as  described  below.  If,  however,  the 
Combat  Developer  identifies  COIs,  all  the  requirements  outlined 
in  AR  73-1  and  this  Pamphlet  apply,  i.e  a  TIWG  must  be  formed,  a 
TEMP  developed.  Independent  Evaluations  must  be  conducted,  etc. 

f.  The  key  factor  in  determining  the  level  of  T&E  conducted 
on  a  modification  is  the  presence  or  absence  of  COI  as  determined 
by  the  CBTDEV/ Functional  Proponent.  The  Combat  Developer 
Coordination  Checksheet  is  required  to  be  completed  on  all 
modification  efforts,  however  for  those  Modification  with  the 
CCB/CMgr  as  decision  authority,  no  formal  coordination  with  the 
CBTDEV/Functional  Proponent  is  required. 

7-3.  Systems  Modification  TEMPs 

a.  TEMPS  for  modification  programs  should  be  separate, 
"stand-alone"  documents.  Use  of  the  original  system's 
development  TEMP  is  inappropriate,  as  the  modification  effort 
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will  have  unique  operational  and  technical  issues.  The  TEMP 
should  address  only  the  elements  of  the  parent  system  that  are 
effected  or  that  have  an  interface  with  the  modification  effort. 

b.  When  several  modifications  are  being  conducted  on  one 
system,  consolidation  of  the  testing  effort  is  desirable.  One 
comprehensive,  consolidated  TEMP  should  be  prepared  outlining  the 
planned  T&E,  and  all  data  should  be  shared  to  ensure  maximum 
efficiency. 

7-4.  Systems  Modification  and  Post  Deployment  Software  Support 
(PDSS)  Test  and  Evaluation. 

a.  Test  and  evaluation  will  be  conducted  on  all  modifications 
and  Post  Deployment  Software  Support  (PDSS) .  The  scope  and  type 
of  T&E  will  vary  for  each  Modification/PDSS  and  will  be 
documented  in  accordance  with  this  pamphlet. 

b.  The  following  criteria  will  be  used  to  determine  the  T&E 
conducted  on  Modification/PDSS: 

(1)  Upon  initiation  of  an  Modification  or  PDSS,  the 
program  manager  will  determine  if  formal  coordination  with  the 
combat  developer  or  functional  proponent  is  required.  When 
formal  coordination  is  required  and  the  CBTDEV  or  FP  indicates 
that  there  are  critical  operational  issues  associated  with  the 
Modification  or  PDSS,  the  program  manager  will  charter  a  TIWG. 

The  TIWG  will  determine  the  type  and  amount  of  testing  that  is 
required  to  be  performed  on  the  Modification/PDSS.  This  testing 
will  be  documented  in  the  TEMP.  The  level  of  program  management 
will  be  categorized  based  on  the  same  criteria  used  for  new 
starts,  and  the  decision  authority  or  TEMP  approval  process  will 
be  consistent  with  the  categorization.  An  OA/AOA  will  be 
prepared  by  the  independent  operational  evaluator,  and  a 
developmental  evaluation  or  assessment  report  will  be  prepared  to 
support  the  decision  review  for  approval  of  procurement  and 
application. 

(2)  When  the  formal  coordination  with  the  combat  developer 
or  functional  proponent  is  not  necessary,  or  when  the  combat 
developer  or  functional  proponent  indicates  that  there  are  no 
critical  operational  issues  associated  with  the  Modification  or 
PDSS,  T&E  will  be  conducted  as  appropriate,  documented  and 
included  in  the  program  manager's  package  provided  at  the 
decision  review.  A  TIWG,  a  TEMP  and  an  operational  evaluation 
are  not  required  in  this  case.  However,  the  program  manager  will 
consult  the  MATDEV,  Independent  Developmental  Evaluator  or 
Assessor,  to  determine  if  coordination  is  required  with  the 
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independent  developmental  evaluator  or  assessor.  When 
coordination  is  required,  an  independent  developmental  evaluation 
or  assessment  will  be  performed  unless  waived  by  the  independent 
developmental  evaluator  or  assessor.  Waiver  of  the  requirement 
for  a  developmental  evaluation/assessment  may  occur  if  the 
checksheet  indicates  that  coordination  is  not  required  or  if  the 
developmental  evaluator /assessor,  after  review  of  the  package, 
determines  that  evaluation  or  assessment  is  not  needed.  In  this 
case,  formal  notification  to  the  program  manager  will  be  made  by 
the  independent  developmental  evaluator/assessor . 

c.  When  a  block  modification  is  being  tested  or  when  testing 
for  more  than  one  Modification  or  type  of  PDSS  is  actively  being 
conducted  on  a  system,  and  are  planned  to  be  integrated  into  the 
same  end  item,  efforts  will  be  integrated  so  that  maximum  T&E 
efficiencies  are  achieved.  One  TEMP  (in  those  cases  where  a  TEMP 
is  required)  shall  be  prepared  integrating  the  T&E  required  into 
one  comprehensive  master  plan.  The  TIWG  members  shall  plan  and 
conduct  T&E  so  that  maximum  efficiencies  in  assessing  specific 
aspects  of  the  various  Modif ication/PDSS  can  be  achieved.  In 
cases  where  integration  is  not  practical  (e.g.,  different  program 
manager  agencies) ,  every  effort  shall  be  made  to  assure  that 
system  integration  testing  of  all  Modif ication/PDSS  is  performed. 

d.  Test  and  evaluation  of  modifications  and  PDSS  require  both 
developmental  and  operational  testing.  The  purpose  of  this 
testing  is  to  determine  the  viability  and  adequacy  of  the  change 
and  to  determine  if  the  change  was  achieved  without  degradation 
to  the  system,  other  components,  or  interface  equipment. 
Modification  involves  reconfiguration  of  a  fielded  item  to 
provide  new  or  enhanced  capability,  extend  the  useful  life  of  the 
system,  improve  safety  or  readiness,  or  reduce  operation  support 
cost.  Modifications  includes  Product  Improvement  Programs  (PIPs) 
which  are  not  preplanned. 

e.  Modifications  are  normally  necessitated  because  some 
(jevelopment  (e.g,  technological  breakthrough,  doctrinal  change, 
revised  threat  estimate)  has  occurred  after  the  original  system 
configuration  was  fixed.  Documentation  and  testing  are  reduced 
from  new  system  development,  as  are  milestone  reviews,  because 
both  documents  and  decision  issues  are  focused  on  the  areas  of 
change.  Generally,  if  the  performance  envelope  changes 
significantly,  requirements  documents  are  required,  and  the 
associated  TEMP  and  OT&E  cycles  apply.  However,  if  the 
modification  is  focused  only  on  reduced  operating  costs,  minimal 
operational  testing  is  performed  to  ensure  no  degradation  of 
overall  system  performance.  The  independent  operational 
evaluator  must  draw  upon  military  expertise,  materiel  acquisition 
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knowledge,  and  current  Army  policy  to  make  a  recommendation  to 
the  TIWG.  The  evaluator,  in  coordination  with  the  ODCSOPS  staff 
officer,  informs  DOTE  of  the  modification  if  the  system  has  been 
designated  for  DOTE  oversight. 

f.  The  product  of  T&E  is  information  which  Army/DOD 
leadership  needs  to  make  informed  decisions.  Testing  involves 
costs  in  terms  of  time,  resources  and  effort.  The  primary  reason 
for  testing  i's  to  obtain  information  which  is  unknown  and  can 
only  be  learned  by  testing.  Evaluation  is  a  continuous  process 
which  will  be  accomplished  on  all  modifications.  In  this  light, 
testing  and  evaluation  of  modifications  will  be  tailored  to  the 
specific  system  requiring  modification.  The  extent  and  type  of 
T&E  varies  for  each  modification.  Testing  and  evaluation  will  be 
performed  and  documented  in  accordance  with  DOD  5000  series 
guidance,  AR  73-1,  AR  70-1  and  other  applicable  T&E  guidance. 

g.  The  purpose  of  developmental  T&E  is  to  determine  the 
safety,  viability  and  adequacy  of  the  modification  and  to 
determine  if  performance,  system  and  critical  technical 
parameters  objectives  for  the  modification  have  been  met.  The 
purpose  of  operational  testing  and  evaluation  is  to  determine  if 
the  modification  is  operationally  effective  and  suitable  for  use. 
Both  developmental  and  operational  T&E  results  may  be  used  to 
ensure  that  the  proposed  modification  is  cost  effective  and 
economical  relative  to  other  alternatives. 

h.  Several  factors  will  determine  the  extent  of  T&E  which 
will  be  required  for  a  modification.  To  determine  the  degree  of 
T&E  needed.  An  abbreviated  T&E  procedure  normally  requires  no 
testing  and  limited  evaluation.  The  following  criteria  will  be 
used  to  determine  requirements  for  abbreviated  T&E  of 
modifications. 

(1)  When  a  program  sponsor  has  determined  that  the 
abbreviated  T&E  process  can  be  used,  the  TIWG,  TEMP  and 
inependent  operational  evaluation  can  be  omitted. 

(2)  An  independent  developmental  evaluation  or  assessment 
may  be  required  for  modifications  utilizing  the  abbreviated  T&E 
process.  The  criteria  used  to  establish  the  need  for  an 
independent  developmental  evaluation  is  as  follows: 

(a)  The  program  sponsor  completes  the  "Program 
Sponsor  &  Independent  Developmental  Evaluator /Assessor 
Coordination  Checksheet"  listed  at  Figure  7-2.  If  the  answer  to 
any  question  on  the  checksheet  is  "NO",  then  formal  coordination 
with  the  independent  developmental  evaluator  and  a  independent 
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developmental  evaluation  are  not  required. 

(b)  If  the  answer  to  any  question  on  the  checksheet  is 
"YES”  or  "MAYBE"  then  the  program  sponsor  must  send  a  complete 
modification  data  package  to  the  independent  developmental 
evaluator.  Upon  receipt  of  the  Modification  Data  Package  the 
independent  developmental  evaluator  will  perform  a  review  and 
make  a  determination  whether  the  modification  should  have  an 
independent  developmental  evaluation.  This  determination  should 
be  made  considering  such  factors  as  risk  associated  with  the 
technology,  safety,  environmental  issues,  impact  on 
survivability/vulnerability,  MANPRINT  concerns,  or  any  other 
factor  that  the  independent  developmental  evaluator  considers 
appropriate. 

(c)  The  independent  developmental  evaluator  will 
provide  formal  notification  to  the  program  sponsor  informing  him 
whether  an  evaluation  will  be  performed. 

(d)  When  the  independent  developmental  evaluator 
determines  that  an  evaluation  is  required,  an  lEP  shall  be 
prepared  outlining  the  required  testing  and  evaluation  by  the 
independent  developmental  evaluator,  and  a  copy  of  the  lEP  should 
be  provided  to  the  program  sponsor  to  facilitate  program 
planning.  The  need  for  an  evaluation,  however,  does  not 
necessarily  mean  that  testing  also  is  required.  An  evaluation 

of  existing  data  can  be  performed  or,  alternatively,  dedicated 
testing  may  be  required  to  address  the  technical  concerns. 

7-5.  Post  Deployment  Software  Support  (PDSS)  Requirements 

a.  Software  changes  to  deployed  systems  are  generated  because 
of  latent  defects,  doctrinal  requirements,  threat  changes, 
weapon/munitions  upgrades,  interoperability  requirements,  product 
improvements  and  new  system  functions.  Change  requests  are 
normally  generated  by  the  using  agency,  CBTDEy,  or  Functional 
Proponent  and  forwarded  for  approval,  prioritization  and 
implementation.  Changes  are  generated  in  accordance  with  AR  25- 
3,  AR  71-9,  and  DA  Pamphlet  25-6. 

b.  Emergency  changes  frequently  need  to  be  released  to  the 
field  within  48  hours.  In  this  case,  the  T&E  team  needs  to 
respond  quickly  to  ensure  that  the  software  changes  are  made  and 
adequately  retested.  Because  changing  software  to  fix  a  problem 
can  introduce  additional  problems,  retesting  must  be  done  along 
with  testing  the  fix.  It  is  permissible  for  formal  documentation 
updates  to  catch  up  later.  All  changes  to  the  software  must  be 
identified  and  controlled  by  the  configuration  management 
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function. 

c.  While  all  changes  for  systems  under  development  must 
undergo  validation  and  verification  testing,  emergency  changes  to 
deployed  systems  may  not  require  formal  developmental  testing  or 
operational  testing.  However,  all  emergency  changes  will  undergo 
formal  testing  with  the  next  planned  updates.  The  system  or 
operation  manager,  with  the  concurrence  of  the  system  user,  may 
only  be  capable  of  performing  limited  testing  of  emergency 
software  corrections  prior  to  granting  release. 

7-6.  Post  Deployment  Software  Support  (PDSS)  Test  and  Evaluation 

a.  PDSS  consists  of  modifications  and  maintenance  of  software 
in  fielded  systems  or  systems  to  be  fielded  after  MS  III 
decision.  The  development  of  a  change  follows  the  basic  life 
cycle  process  as  a  new  system;  however,  the  scope  and  resources 
involved  may  be  reduced.  The  PDSS  environment  generally  produces 
many  small  changes  over  a  period  of  time  rather  than  a  few  large 
changes.  The  PDSS  organization  must  be  able  to  coordinate  and 
phase-in  these  many  changes  into  a  few  formal  software  releases 
to  minimize  disruption  of  fielded  systems.  Differences  in  the 
magnitude  and  timing  of  software  changes  must  be  considered  in 
identifying  the  scope  of  T&E  required  and  the  extent  of  T&E  team 
involvement.  The  concept  of  block  changes  and  block  releases 
must  be  integrated  into  the  T&E  planning. 

b.  PDSS  test  and  evaluation  refers  to  changes  made  to  the 
system  after  first  unit  equipped  (FUE)  or  first  site  fielded. 
These  changes  are  within  the  authority  of  the  system  manager  or 
MATDEV.  Test  and  evaluation  of  changes  during  PDSS  and  software 
development  activities  are  contained  in  Part  Seven. 

c.  PDSS  requires  some  flexibility  in  development,  testing, 
and  evaluation,  in  order  to  respond  to  the  changing  environment. 
Often,  new  requirements  have  date-driven  implementation 
milestones  which  must  be  accommodated  to  preclude  adverse  impact 
on  system  performance.  Emergency  changes  which  must  be  completed 
within  48  hours  may  be  required.  The  T&E  team  must  be  responsive 
and  is  an  integral  element  of  effective  PDSS  support  and  fielding 
of  quality  products. 
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1.  This  checksheet  is  to  be  used  by  the  Program  Sponsor  to 
determine  if  coordination  with  the  CBTDEV/Functional  Proponent  is 
required.  The  purpose  of  the  checksheet  is  to  provide  a 
simplified  method  of  determining  whether  a  proposed  MC  has 
CBTDEV/Functional  Proponent  impact/ interest  in  order  to  ease  the 
administrative  burden  of  coordination  between  the  Program  Sponsor 
and  CBTDEV/Functional  Proponent. 

2.  The  checksheet  determines  whether  CBTDEV/Functional  Proponent 
interest  is  present,  based  on  evaluation  of  the  following 
conditions  regarding  the  proposed  MC: 

a.  Implementation  of  the  proposed  change  affects  the  execution 
of  another  funded  program  or  requires  competition  in  the  PPBES  to 
acquire  funding. 

b.  Calls  for  changes  to  the  existing  performance  capability  of 
the  system  beyond  the  bounds  of  the  existing  requirements 
document  as  amended  by  subsequent  decisions. 

c.  Affects  the  BOIP,  QQPRI,  or  TOiEs  associated  with  the 
system. 

d.  Affects  the  way  the  system  Is  operated,  employed  and 
maintained. 

e.  Alters  MANPRINT  requirements  (manpower,  personnel, 
training,  human  factors  engineering,  system  safety,  health 
hazards) . 

f.  Has  safety  implications. 

3.  The  Program  Sponsor  or  his  designated  representative  is 
responsible  for  evaluating  the  change  against  the  checksheet, 
answering  its  questions,  and  verifying  its  correctness  by 
providing  the  information  required  and  signing  at  the  bottom.  The 
form  is  then  enclosed  in  the  MC  package. 

4.  If  any  of  the  questions  on  the  checksheet  can  be  answered  "YES 
or  "MAYBE",  then  the  entire  MC  package  must  be  coordinated  with 
the  CBTDEV/Functional  Proponent  prior  to  implementation  of  the 
MC.  Conversely,  if  all  the  checksheet  questions  can  be  answered 
"NO",  CBTDEV/Functional  Proponent  concurrence  is  necessary  only 
for  those  MCs  which  require  Program  Sponsor  level  approval  or 
higher . 

5.  The  following  is  additional  guidance  concerning  answering  the 

Figure  7-1  Combat  Development/Functional  Proponent  Coordination 
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checksheet  questions. 

a.  Answer  each  question  "YES”,  "NO",  or  "MAYBE." 

b.  Use  of  the  "MAYBE"  answer  is  appropriate  if  there  is  doubt 
that  the  answer  is  clearly  "YES"  or  "NO" . 

c.  Consult  AR  73-1,  Testing  and  Evaluation,  for  any 
clarification  of  question  "a". 

d.  Consult  the  MNS,  ORD,  or  other  basic  requirements  document 
for  the  system  in  answering  question  "b" . 

e.  Questions  -d"  through  "h"  are  derived  from  requirements 
found  in  AR  71-2,  BOIP/QQPRI  and  also  have  MANPRINT  implications. 

f .  Local  reproduction  of  the  checksheet  is  authorized. 

6.  Discussions  between  the  Program  Sponsor  and  CBTDEV/ Functional 
Proponent  representatives  are  encouraged  to  prevent  the  checklist 
from  unnecessarily  elevating  the  decision  level  of  an  MC.  Any 
agreements  to  retain  the  MC  at  a  lower  decision  level  will  be 
recorded  by  the  Program  Sponsor  and  become  part  of  the  MC  file. 
Would  application  of  this  MC  proposal  to  the  system/end  item — 

a.  Require  some  form  of  user  testing  (As  opposed  to 

durability  or  reliability  testing)?  (YES _ NO  MAYBE _ ) 

b.  Require  funding  diversions  which  will  impact  the 

performance/completion  of  other  funded  aspects  of  this  or  other 
programs,  or  rely  on  successful  competition  for  funds  not 
currently  programed?  (YES _ NO _ ^MAYBE _ ) 

c.  Affect  the  system's  operational  characteristics, 

performance,  or  tactical  employment  in  a  way  which  will  impact 
the  user?  (Example  —  weight,  speed,  rate  of  climb  or 
acceleration,  transportability,  lethality,  center  of  gravity)? 
(YES _ NO  MAYBE _ ) 

d.  Alter  the  number  or  type  of  supporting  equipment  for  this 
system,  including  Associated  Support  Items  of  Equipment  (ASIOE) 
or  Test,  Measurement,  and  Diagnostic  Equipment  (TMDE)? 

(YES _ NO  MAYBE _ ) 

e.  Alter  the  number  or  type  (require  a  new  or  different  MOS) 
of  crew  and/or  support  personnel  required  to  operate  or  maintain 
the  equipment,  or  require  the  formulation  of  additional 

Figure  7-1  Combat  Development/Functional  Proponent  Coordination 
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operational  or  support  units?  (YES _ NO  MAYBE _ ) 

f.  Affect  human  factors  which  will  alter  internal  operating 

procedures  or  skills  with  the  system  to  such  a  degree  that 
training/ operating  instructions  for  operators  or  field  operations 
must  be  changed?  (Examples,  adding  a  radar  warning  system  or 
changing  gunnery  procedure) .  (YES _ NO  MAYBE _ ) 

g.  Alter  the  maintenance  procedures  with  regard  to  time  and 
equipment  required,  disassembly /assembly  sequence  and/or  trouble 
shooting  instructions  to  such  a  degree  that  training/operating 
instructions  for  maintainers  must  be  changed? 

(YES _ NO _ ^MAYBE _ ) 

h.  Require  corresponding  changes  to  simulators  or  training 

devices  for  the  system,  or  vice  versa?  (YES _ NO  MAYBE _ ) 

i.  Have  safety  implications  (i.e.,  correct  and  existing 
safety  problem  or  create  a  potential  safety  problem)? 

(  YES _ NO  MAYBE _ ) 


ACTIVITY _ SIGNATURE _ 

DSN _ PRINTED  NAME  &  RANK _ 

(Provide  explanatory  comments,  as  required) . 


Figure  7-1  Combat  Development/Functional  Proponent  Coordination 
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1.  This  checksheet  will  the  Program  Sponsor  in  determining 
whether  an  MC  proposal  requires  formal  coordination  with  the 
Independent  Developmental  Evaluator/Assessor  (IDE/A) .  If  the 
answer  to  any  question  is  "YES”  or  "MAYBE",  the  Program  Sponsor 
must  send  a  complete  MC  package  to  the  IDE/ A  so  that  the  need  to 
conduct  a  formal  evaluation/assessment,  system  safety 
confirmation,  and  developmental  test  planning  can  be  determined. 
If  all  answers  are  "NO",  formal  coordination  is  not  required. 

2.  Would  approving  this  change  to  the  system/ end  item  : 

a.  Significantly  alter  the  technical  characteristics 

of  the  system/ end  item.  (YES _ NO  MAYBE _ ) 

b.  Affect  the  systems  developmental  characteristics  or 
performance  or  alter  the  performance  of  a 
subsystem. 

(YES _  NO  _ ^MAYBE _ ) 

c.  Employ  new  or  not  fully  proven  technology. 

(YES _ NO _  MAYBE _ ) 

d.  Affect  RAM  characteristics  of  the  system/ end  item. 

(YES _ NO _ ^MAYBE _ ) 

e.  Affect  the  survivability  or  vulnerability  of  the 

system  or  end  item.  (YES _ NO  MAYBE _ ) 

f.  Have  safety  implications  (i.e.,  correct  an  existing 
problem  or  create  a  potential  safety  problem) . 

(YES _  NO _ ^MAYBE _ ) 

g.  Impact  other  MANPRINT  considerations  (manpower, 
personnel,  training,  HFE,  health  hazards) . 

(YES  _ NO _  MAYBE _ ) 

h.  Effect  the  environment.  (YES _  NO _  MAYBE _ ) 


Figure  7-2  Program  Sponsor  and  Independent  Evaluation/ 
Assessor  Coordination  Checksheet 
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Chapter  8 

Test  Integration  Working  Group 


Section  I 
Introduction 


8-1.  General 

The  TIWG  has  been  established  as  the  forum  to  effect  coordination 
and  solve  routine  problems  in  the  Test  and  Evaluation  process. 

By  bringing  together  the  many  agencies  involved  in  the  T&E 
process,  the  TIWG  chairman  can  brief  the  current  status  of  the 
program,  the  future  events  that  will  take  place,  and  emphasize 
the  work  that  must  be  done  by  which  agencies  to  ensure  that  a 
well  orchestrated  T&E  program  is  being  conducted. 

8-2.  TIWG  Team 

The  TIWG  is  a  team  of  highly  qualified  members  representing  their 
respective  organizations  who  meet  to  plan  the  necessary  testing 
and  the  attendant  evaluations.  Through  the  intense  efforts  of 
this  team  the  planning,  scheduling,  resourcing,  and  actual 
testing  can  be  accomplished.  The  team  effort  establishes  a  T&E 
program  that  will  address  whether  the  risks  of  developing  and 
producing  required  systems  are  within  acceptable  and  safe 
parameters.  Actual  testing  or  the  availability  of  existing  and 
directly  applicable  test  data  will  assure  that  all  technical  and 
operational  characteristics  and  issues  are  measured  or  assessed 
as  comprehensively  as  possible. 

8-3 .  Purpose 

The  primary  purposes  of  the  TIWG  are  to  optimize  the  use  of 
appropriate  T&E  expertise,  instrumentation,  targets,  facilities, 
simulations,  and  models  to  implement  test  integration,  thereby 
reducing  costs  to  the  Army;  integrate  test  requirements;  concur 
in  the  TEMP  as  the  first  step  in  the  TEMP  approval  process; 
mutually  resolve  cost  and  scheduling  problems;  provide  a  forum  to 
assist  those  responsible  for  T&E  documentation  and  execution:  and 
ensure  that  T&E  planning,  execution,  and  reporting  are  directed 
toward  common  goals. 

8-4.  Goals 

TIWG  goals  are  to  develop  a  mutually  agreeable  test  program  that 
will  provide  the  necessary  test  data  for  evaluations;  to  provide 
for  development,  staffing,  coordination  and  approval  of  all 
required  T&E  documentation;  establish  the  necessary  subordinate 
working  groups  (subgroups)  to  address  related  T&E  issues;  assure 
all  participants  have  the  opportunity  to  be  involved  and  not  be 
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excluded;  establish  and  manage  the  corrective  action  process; 
participate  in  technical  and  operational  test  readiness  reviews 
(TTRR  and  OTRR)  and  to  support  the  continuous  evaluation  (CE)  and 
integrated  T&E. 


Section  II 
Objectives 


8-5.  TIWG  Forum 

TIWGs  provide  a  forum  in  which  designated  representatives  of  each 
member  organization  can  discuss  freely  their  test  requirements; 
mutually  resolve  cost  and  scheduling  problems;  and  assure  that 
T&E  planning,  execution,  and  reporting  are  directed  towards  a 
common  goal.  T&E  coordination  among  all  members  of  the 
acquisition  team  (AT)  is  accomplished  through  the  TIWG.  To  this 
end,  TIWG  members  are  members  of  the  AT  and  remain  a  principal 
active  working  group  throughout  the  system  acquisition  process. 

8-6.  TIWG  Meetings 

TIWG  meetings  encompass  activities  such  as:  the  airing  of 
substantive  developmental  and  operational  issues;  briefings  by 
special  interest  activities  (e.g.,  safety,  environmental, 
software) ;  coordination  of  schedules,  test  plans.  Test  & 
Evaluation  Master  Plan  (TEMP),  etc.;  and  identification  of 
problems  and  resolution  of  issues. 

8-7.  The  TIWG  Charter 

The  TIWG  is  chartered  to  perform  the  function  of  structuring  the 
test  and  evaluation  (T&E)  program  and  integrating  the  various  T&E 
and  data  requirements.  It  is  chaired  by  the  PM/MATDEV  and  is 
constituted  of  qualified  T&E  representatives,  with  the  authority 
to  speak  and  sign  for  their  parent  organization.  TIWG  members 
are  obligated  to  participate  in  TIWG  meetings  unless  the  agenda 
does  not  include  topics  where  they  have  a  direct  interest. 

8-8.  Establishment  of  the  TIWG 

The  TIWG  is  established  prior  to  Milestone  I  (MS  I)  by  the 
PM/MATDEV  (or  appropriate  acquisition  team  if  the  PM  is  not 
designated)  after  receipt  of  the  approved  Mission  Need  Statement 
(MNS) .  This  will  be  in  sufficient  time  to  assist  in  finalizing 
the  critical  operational  issues  and  criteria  for  decision 
authority  approval  at  MS  I,  and  will  facilitate  early  development 
of  the  TEMP  and  the  T&E  portions  of  the  Request  for  Proposal 
(RFP)  and  supporting  documentation.  Close  coordination  among  the 
TIWG  members  must  be  effected  in  a  timely  manner  in  order  to 
optimize  schedules  and  costs  and  preclude  duplication  or  voids  in 
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the  acquisition  test  cycle. 

8-9.  CE 

The  TIWG  supports  continuous  evaluation  (CE)  by  accomplishing 
earlier,  more  detailed,  and  continuing  T&E  documentation, 
planning,  integration,  and  sharing  of  data  from  all  testing.  T&E 
documentation  should,  if  possible,  not  be  published  without  first 
allowing  the  principal  TIWG  members  to  review  (not  necessarily 
with  any  form  of  approval  authority)  the  document  thoroughly. 

This  process  will  ensure  that  accurate  T&E  documentation  will  be 
published. 

8-10.  TIWG  Members 

TIWG  members  ensure  transmittal  and  review  of  test  incidents  and 
related  reports,  described  in  Chapter  17. 


Section  III 
Composition 


8-11.  TIWG  Participants 

The  TIWG  is  divided  into  two  categories  of  participants, 
"principal"  and  "associate"  members.  TIWGs  are  chaired  by  a  PM 
representative.  Generally,  the  TIWG  consists  of  the  following 
principal  members: 

a .  PM/MATDEV 

b.  Combat  Developer /Functional  Proponent,  TRADOC  for  AR  70 
systems,  any  MACOM  or  DA  staff  section  for  AR  25  systems. 

c.  Developmental  Tester,  TECOM  for  AR  70  systems  and  TECOM, 
ISEC,  or  AMC/AMC  MSC  for  AR  25  systems. 

d.  Developmental  Independent  Evaluator/Assessor,  AMSAA  or 
TECOM  for  AR  70  systems  and  AMSAA,  TECOM  or  ISEC  for  AR  25 
systems . 

e.  Operational  Tester,  TEXCOM  for  AR  70  systems  and  TEXCOM, 
INSCOM,  any  MACOM  or  ISC/ ISEC  for  AR  25  Systems. 

f.  Operational  Independent  Evaluator,  OEC  for  both  AR  70  and 
25  systems. 

g.  Logistician,  AMSAA  for  AR  70  systems  and  AMSAA  or  the 
Developmental  IE  for  AR  25  systems. 
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h.  Survivability/Lethality  Analysis  Directorate  (SLAD) . 
Responsible  for  determining  the  survivability/ lethality  and 
vulnerability  (SLV)  of  Army  systems  to  the  full  spectrum  of 
battlefield  threats. 

i.  Threat  Integrator,  Threat  Systems  Officer  (TSO)  from  the 
TRADOC  school  initiating  the  requirement  for  AR  70  systems.  A 
threat  integrator  is  generally  required  for  theater/tactical  AR 
25  systems.  The  threat  integrator  is  a  principal  member  of  the 
TIWG  only  when  the  system  being  acquired  is  intended  to  defeat  a 
specific  threat  system/s. 

j.  Trainer.  The  trainer  is  a  principal  member  of  the  TIWG 
only  when  the  CBTDEV  is  a  separate  agency  from  that  which  will  be 
providing  training  for  the  program,  e.g.  Special  Operations 
Forces  is  the  CBTDEV,  Army  Infantry  School  is  the  Trainer. 

8-12.  Other  TIWG  Members 

Additional  agencies  which  can  be  principal  members  of 
the  TIWG  include: 

a.  The  Army  Command  Control  System  (ACCS)  systems  engineer 
for  any  ACCS  component  system  or  equipment  which  has  one  or  more 
interfaces. 

b.  The  PM  Smoke/Obscurants  for  all  systems  which  rely  on 
electro-optical  propagation  and  are  susceptible  to  aerosol 
countermeasures . 

c.  Military  Traffic  Management  Command  Transportation 
Engineering  Agency  if  transportability  engineering  analysis  of 
"problem  items,"  in  accordance  with  AR  70-47,  has  identified  any 
transportability  issues. 

d.  Joint  Tactical  Communications,  Command,  and  Control  Agency 
for  tactical  C3  intelligence  systems. 

/ 

e.  U.  S.  Army  Defense  Ammunition  Center  and  School  (USADACS) 
when  ammunition  restraint  system  procedures  need  to  be  developed 
for  military  vehicles  so  that  appropriate  testing  can  be 
performed  on  design  tiedown  points. 

f.  PMs/MATDEVs  from  other  programs  that  are  being  developed 
concurrently  as  part  of  a  single  system.  This  can  occur  when  two 
vehicles  or  major  subsystems  are  being  developed  concurrently  by 
two  different  organizations  as  part  of  one  program. 

g.  Representatives  from  the  Army  Research  Laboratory  (ARL) . 


8-4 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


h.  Representatives  of  other  services  for  multiservice 
acquisitions. 

8-13.  TIWG  Associated  Members 

The  associate  members  of  the  TIWG  may  consist  of  any 
representative  who  provides  a  needed  supportive  role  to 
adequately  address  all  necessary  T&E  requirements,  and  support 
the  subordinated  working  groups.  These  associate  members  can 
include  the  Integrated  Logistics  Management  Support  Team  (ILSMT) , 
the  international  materiel  evaluation  representative,  the 
contractor  (when  appropriate),  PM  Instrumentation,  Targets  and 
Threat  Simulators  (ITTS) ,  and  those  commands/activities  which 
serve  in  a  monitor's  role  (e.g..  The  Surgeon  General  (TSG) 
representative  for  health  aspects  associated  with  system 
testing/use. 

8-14.  PM/MATDEV 

The  PM/MATDEV  will  make  the  initial  determination  of  the  TIWG 
members  which  will  be,  at  a  minimum,  those  members  identified  in 
paragraph  8-9. 

8-15.  Multiservice  Acquisition  Programs 

Multiservice  Acquisition  Programs  with  Army  lead  will  have  the 
same  Army  TIWG  membership  as  an  Army  unique  acquisition  program. 
Participating  services  will  determine  their  membership 
requirements  and  those  will  be  documented  in  the  TIWG  charter. 
Multiservice  programs  with  Army  participation  (not  lead)  will 
have,  as  a  minimum,  representatives  from  the  PM/MATDEV, 

CBTDEV/ Functional  Proponent,  TIE,  and  lOE.  If  any  Army  unique 
testing  is  planned,  the  appropriate  test  agency  shall  also  be 
represented.  As  in  all  cases,  TIWG  membership  is  documented  in 
the  Charter. 


Section  IV 
TIWG  Subgroups 


8-16.  TIWG  Subgroups 

Essential  to  the  TIWG  process  is  the  performance  of  specialized 
tasks  assigned  to  subordinate  working  groups.  The  subgroups  are 
necessary  to  define  the  details  of  the  T&E  program,  handle  the 
interfaces  with  other  disciplines,  prepare  for  testing,  and 
develop  supporting  T&E  documentation.  Additionally,  the 
subgroups  are  required  to  coordinate  and  jointly  develop  T&E 
parameters  and  identify  corrective  actions.  When  possible  the 
T&E  charter  will  delineate  the  planned  subgroups.  In  some  cases 
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the  subgroups  may  need  to  establish  their  own  work  groups. 

8-17.  TIWG  Subgroup  Charters  .  . 

The  TIWG  will  charter,  as  necessary,  the  subgroups  identified 
below.  Other  subgroups  may  be  chartered  as  appropriate. 

a.  RAM  Working  Group  (RAMWG) .  Co-chaired  by  the  ip^TDEV  and 
CBTDEV,  this  group  will  address  all  RAM  issues  including  failure 
definition/scoring  criteria,  RAM  Rationale  Annex,  and  Data 
Collection.  The  technical  and  operational  evaluators,  as  a 
minimum,  participate  in  this  subgroup.  Further  details  of  this 
subgroup  are  contained  in  AR  702-3. 

b.  The  Supportability  T&E  Working  Group  (STEWG)  Subgroup  is 
chaired  by  the  PM/MATDEV  ILS  manager.  The  purpose  of  this 
subgroup  is  to  provide  coordination  of  the  TIWG  activities  with 
the  Integrated  Logistic  Support  Management  Team  (ILSMT) .  Topics 
to  be  coordinated  will  include  all  supportability  test  issues, 
test  reguirements ,  and  logistic  demonstration  reguirements 
contained  in  the  TEMP.  Further  details  of  this  subgroup  are 
Contained  in  AR  700-127. 


Section  V 
Interface  Groups 


8-18.  Other  Working  Groups  ^ 

There  are  many  related  disciplines  which  have  a  close  tie  with 
the  TIWG  and  their  working  group  activities  occur  concurrently 
and  often  combined  with  the  activities  of  the  TIWG.  The 
communication  lines  between  these  groups  with  the  TIWG  must  be 
clear  and  allow  for  the  transfer  of  information  to  enhance  the 
progression  of  work  for  all  disciplines.  Some  of  these  closely 
related  subgroups  are  as  follows: 

a.  The  Threat  Coordinating  Subgroup  is  chaired  by  the  threat 
integrator  member  of  the  TIWG.  The  purpose  of  this  subgroup  is 
to  develop,  coordinate  and  maintain  Threat  Test  Support  Package 
(TTSP) . 


b.  Operational  Test  Readiness  Review  Working  Group  (OTRRWG) . 
The  OTRRWG  evaluates  the  system's  readiness  to  enter  OT. 
Membership  includes  the  PM/MATDEV,  OT,  and  lOE.  For  additional 
information  on  the  OTRRWG,  see  Part  V  of  this  Pamphlet. 

c.  Developmental  Test  Readiness  Review  Working  Group 
(DTRRWG) .  The  DTRRWG  evaluates  the  system's  readiness  to  enter 
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developmental  test.  Membership  includes  the  PM/MATDEV,  DT  and 
Developmental  Independent  Evaluator  (DIE) .  For  additional 
information  on  the  DTRRWG,  see  Part  IV  of  this  Pamphlet. 

d.  Data  Authentication  Group  (DAG) .  Data  Authentication 
Group  (DAG) .  The  operational  tester  determines  the  need  for  a 
DAG.  The  DAG  is  chaired  by  the  operational  tester  with 
representatives  from  required  areas  of  expertise  in  attendance. 

It  meets  while  tests  are  being  conducted  to  ensure  timely 
exchange  of  data  among  all  participating  agencies /commands  and  to 
build  a  factual  data  base  by  assisting  in  data  reduction,  data 
analysis,  and  the  investigation  of  problems  surfaced  in  test 
data.  The  group  is  formed  when  the  evaluation  of  systems  require 
complex  data 

collection/instrumentation.  Its  members  may  also  comprise  the 
membership  of  the  RAM  scoring  and  assessment  conferences,  and  the 
DAG  functions  may  be  conducted  concurrently  with  those  of  the 
other  two  conferences  when  appropriate.  Composition  of  the  DAG 
for  an  OT  is  included  in  its  OTP.  For  additional  information  on 
the  DAG,  see  Part  V  of  this  Pamphlet. 

e.  Computer  Resources  Working  Group  (CRWG) .  The  CRWG  is 
established  by  the  PM/MATDEV  after  MS  I  for  each  AR  70-1  system 
to  aid  in  the  management  of  system  computer  resources.  The  CRWG 
assists  in  ensuring  compliance  with  policy,  procedures,  plans  and 
standards  established  for  computer  resources.  Membership 
includes  CBTDEV,  MATDEV,  Developmental  and  Operational  testers, 
DIE,  lOE  and  the  post  deployment  software  support  activity. 
Members  will  actively  participate  in  all  aspects  of  the  program 
including  computer  resources.  For  additional  information  on  the 
CRWG  see  Part  VI  of  this  Pamphlet. 

f .  Integrated  Logistics  Support  Management  Team  (ILSMT) .  The 
ILSMT  is  established  to  coordinate  overall  ILS  planning  and 
execution.  Membership  includes  the  PM/MATDEV,  DT,  OT,  DIE,  lOE, 
Logistician  and  Trainer.  (SEE  AR  700-127) 

g.  MANPRINT  Joint  Working  Group  (MJWG) .  The  MJWG  develops 
the  System  MANPRINT  Management  Plan  (SMMP)  and  coordinates  the 
MANPRINT  program.  Membership  includes  the  PM/MATDEV,  CBTDEV, 
Logistician  and  other  organizations  as  appropriate.  (SEE  AR 
602-2) 


h.  System  Safety  Working  Group  (SSWG) .  The  SSWG  is  chaired  by 
the  PM/MATDEV  and  provides  program  management  with  system  safety 
expertise  and  ensures  enhance  communication  between  all  AT 
members.  Membership  includes  the  PM/MATDEV,  DT,  OT,  DIE,  and 
lOE.  (SEE  AR  385-16) 
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i.  Live  Fire  Test  and  Evaluation  Working  Group  (LFT&EWG) . 
The  LFT&EWG  is  chaired  by  AMSAA  and  is  formed  to  prepare  the 
LFT&E  strategy  and  input  to  the  TEMP.  Membership  typically 
includes  the  system  developer,  the  combat  developer,  the 
independent  evaluators,  vulnerability/ lethality  analysts, 
testers,  the  medical  community,  the  intelligence  community  and 
the  system  contractor  (as  required) . 


Section  VI 
Charter 


8-19.  A  charter  is  required  for  each  TIWG 

The  charter  establishes  the  membership,  scope,  objectives  and 
procedures  of  each  TIWG.  A  sample  format  is  at  Figure  8—1.  The 
formal  TIWG  Charter  is  prepared  by  the  PM/MATDEV  and  coordinated 
with  the  principal  TIWG  members. 

(insert  Figure  8-1) 

Section  VII 
Meetings 

8-20.  Initial  TIWG 

The  initial  TIWG  meeting  should  be  held,  possibly  in  conjunction 
with  a  review  of  the  draft  Operational  Requirements  Document 
(ORD)  or  IMA  system  requirements  document,  to  familiarize  the 
TIWG  members  with  the  preliminary  system  requirements.  This 
meeting  can  be  used  to  support  the  PM  in  developing  the  T&E 
strategy  for  incorporation  into  the  acquisition  strategy,  to 
identify  all  required  TIWG  members,  finalize  the  charter  and  task 
TIWG  members  to  prepare  input  for  the  preliminary  TEMP.  Initial 
draft  input  should  be  submitted  to  the  PM  within  30  calendar 
days. 

8-21.  Notice  of  TIWG  Meeting 

Notice  of  the  initial  TIWG  meeting  should  be  sent  at  least  14 
calendar  days  (2  weeks),  preferably  30  days,  prior  to  the  TIWG. 

A  draft  agenda  should  accompany  the  notice.  The  agenda  should  be 
finalized  with  input  solicited  from  the  TIWG  members. 

8-22.  Strawman  TEMP 

When  circumstances  warrant  (e.g.,  accelerated  acquisition) ,  a 
strawman  TEMP  can  be  prepared  by  the  PM  for  review  and  discussion 
at  the  initial  TIWG.  The  strawman  TEMP  should  be  provided  to  the 
TIWG  members  at  least  30  days  prior  to  the  meeting.  A  strawman 
TEMP  can  be  used  to  facilitate  T&E  strategy  discussions  and 
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development  of  the  preliminary  TEMP. 

8-23.  TIWG  Topics 

Topics  which  need  to  be  addressed  at  the  initial  TIWG  include: 

a.  Introduction  to  the  program.  At  the  initial  TIWG,  it  is 
very  likely  that  attendees  will  be  unfamiliar  with  a  new  program 
and  it  is  necessary  to  familiarize  them  with  all  aspects  of  the 
new  system. 

b.  Acquisition  strategy.  Describe  the  overall  acquisition 
approach  that  will  be  employed,  showing  how  the  results  of  the 
T&E  communities  participation  in  the  early  planning  of  the 
acquisition  strategy  ensures  adequate  T&E  is  integrated  into  the 
overall  program. 

c.  Requirements  documents.  Conduct  a  detailed  review  of  the 
Mission  Need  Statement  (MNS)  and  the  draft  ORD  or  functional 
description  if  available.  This  is  to  familiarize  the  TIWG 
members  with  the  requirements  for  the  new  or  modified  system. 

The  Combat  Developer  or  the  Functional  Proponent,  in  the  case  of 
IMA  systems,  should  conduct  the  review. 

d.  T&E  provisions  in  contracts.  Generally,  contractual 
documentation  has  not  been  prepared  at  this  point,  however  it  is 
important  to  stress  that  a  major  function  of  the  TIWG  members  is 
to  review  contractual  documents  for  T&E  adequacy.  If  there  is  a 
draft  Statement  of  Work  (SOW)  or  Request  for  Proposal  (RFP) ,  it 
is  useful  to  highlight  the  contractual  requirements  for  test  and 
evaluation. 

e.  Strawman  TEMP  review.  If  a  strawman  TEMP  is  prepared 
prior  to  the  initial  TIWG,  time  should  be  allotted' at  the  TIWG  to 
review  all  comments  and  proposed  changes  to  the  TEMP.  If  the 
changes  are  satisfactory  to  the  TIWG  members,  the  TIWG 
Coordination  Sheet  can  be  signed  at  the  meeting  site,  or 
alternatively,  signed  within  some  time  frame  that  is  mutually 
agreeable  to  all  principal  TIWG  members. 

8-24.  TIWG  Actions 

At  the  first  TIWG  the  following  actions  should  occur: 

a.  Provide  a  program/ system  orientation  briefing. 

b.  Review  available  system  requirements  documents  to 
familiarize  TIWG  members  with  preliminary  system  requirements. 

c.  Develop  the  T&E  strategy  for  incorporation  into  the 
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acquisition  strategy. 

d.  Initiate  dialogue  to  define  the  technical  and  operational 
issues  to  be  addressed  in  T&E. 

e.  Detail  the  initial  test  requirements  for  the  respective 
life  cycle  phases  that  will  provide  the  test  data  and  evaluations 
needed  for  each  milestone. 

f.  Task  TIWG  members  to  draft  their  respective  portions  of 
the  TEMP  if  a  strawman  is  not  provided.  If  a  strawman  was 
prepared,  TIWG  member  comments  and  recommended  changes  should  be 
discussed.  Agreement  should  be  reached  on  changes  to  be  made  and 
issues  to  be  resolved. 

g.  Review  and  approve  the  TIWG  charter.  Ensure  all  TIWG 
members  are  identified. 

h.  Establish  required  subgroups. 

i.  Discuss  initial  document  development  requirements. 
Immediate  work  should  proceed  to  develop  the  Critical  Operational 
Issues  and  Criteria  (COIC) ,  the  Safety  Assessment  Report  (SAR) , 
the  Security  Classification  Guide  (SCG) ,  Safety  Release  (SR) ,  and 
Environmental  Impact  Statements  (EIS) .  Completion  of  this 
documentation  facilitates  the  test  process. 

j .  Discuss  the  action  items  assigned  and  develop  a 
tentative  agenda  for  the  next  meeting. 

k.  Record  the  minutes  and  action  items.  After  the  meeting 
the  Chair  will  prepare  the  meeting  minutes  including  the  Action 
Item  List  (AIL) ,  and  distribute  as  agreed  to  at  the  meeting  and 
in  the  TIWG  charter. 

l.  Establish  the  TIWG  minutes  distribution  list  containing 
all  pertinent  information,  e.g.  actual  names,  phone  numbers, 
facsimile  numbers,  and  electronic  addresses. 

8-25.  Follow-on  Meetings 

Follow-on  TIWG  meetings  should  occur  on  a  timely  basis  to 
continue  the  T&E  planning  effort  and  the  development, 
coordination,  and  approval  of  the  required  T&E  documentation, 
especially,  the  TEMP.  The  progress  of  the  test  program  will  be 
addressed  and  subgroups  will  meet  as  appropriate.  As  program 
changes  occur  and  testing  details  are  developed,  program  planning 
modifications  will  be  required.  Discussion  of  issues  should 
continually  occur,  and  issues  which  are  resolved  will  be  closed 
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out  in  the  AIL.  DTRRs  and  OTRRs  will  be  conducted  and  any  issues 
relating  to  test  readiness  should  be  raised  and  resolved  at  the 
TIWG.  Techniques  for  data  collection,  incident  reporting  and 
other  test  peculiar  issues  should  be  fully  coordinated  and 
integrated  within  the  T&E  community.  A  TIWG  can  be  held  at  any 
time  in  a  program  when  it  is  necessary  to  assemble  the  many 
agencies  involved  in  the  T&E  process  for  the  program.  This  can 
occur  when  the  program  is  restructured,  when  an  event  presents  a 
serious  conflict  for  the  next  series  of  tests,  during  a  test  to 
disseminate  information,  or  any  other  time. 


Section  VIII 

General  TIWG  Procedures 


8-26.  TIWG  Meeting 

Announcements  for  a  TIWG  must  be  sent  to  all  TIWG  members  at 
least  14  days,  preferably  30  days,  prior  to  the  commencement  of  a 
TIWG.  The  notice  announcing  the  TIWG  should  include  an  agenda  of 
topics  to  be  discussed  that  includes  TIWG  member  topics. 

8-27.  Unresolved  Issues 

The  TIWG  should  not  submerge  the  airing  of  substantive 
developmental  and  operational  issues  by  a  consensus  process. 
Disagreement  on  matters  of  substance  will  be  elevated  through 
command  channels  to  the  next  higher  level  for  review  and 
adjudication.  Issues  not  resolved  will  be  brought  to  the 
DUSA(OR)  for  resolution.  Policy  and  procedural  issues  should  be 
brought  forward  through  TEMA  for  DUSA(OR)  resolution. 

8-28.  Open  Items 

When  an  agenda  item  is  not  completed  or  resolved  during  a  TIWG 
meeting,  it  is  usually  assigned  to  one  of  the  representatives 
(contingent  upon  acceptance)  for  action,  with  appropriate 
suspense  date.  Open  action  items  become  part  of  the  TIWG  Action 
Item  List  (AIL)  and  are  carried  over  to  the  next  TIWG  agenda 
either  to  verify  that  action  has  been  completed  or  accomplish  the 
necessary  closing  action.  The  action  items  should  be  briefed  as 
the  last  agenda  topic  at  the  TIWG. 

8-29.  TIWG  Minutes 

Minutes  of  each  meeting  are  prepared  by  the  chairperson  and 
distributed  to  each  principal  member  (to  include  those  who  could 
not  attend)  within  10  working  days  of  the  TIWG  meeting.  The 
minutes  document  all  decisions  and  agreements  of  the  TIWG  and 
become  a  part  of  the  official  file.  If  the  minutes  do  not 
adequately  reflect  a  member's  understanding  of  what  was 
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accomplished  at  a  TIWG  meeting,  or  if  a  member  organization's 
position  changes,  this  should  be  brought  to  the  attention  of  the 
chairman  for  correction  or  added  as  an  action  item  to  the  next 
TIWG  agenda  within  2  weeks  after  receipt  of  the  minutes. 
Alternatively,  any  reasonable  period  of  time,  as  agreed  to  by  all 
TIWG  members  and  documented  in  the  Charter,  can  be  used. 

8-30.  Teleconferences 

Consideration  should  be  given  to  conducting  limited  scope  TIWG 
meetings  by  video  teleconference.  Normally  conference  time  is 
limited  to  2  hours.  This  method  is  good  for  disseminating 
information  and  reviewing  comments  requiring  TEMP  changes. 

8-31.  Coordination 

Coordination  on  documents  can  be  done  by  telephone  or  facsimile 
machine. 
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CHARTER  OF  THE  _ * _ 

TEST  INTEGRATION  WORKING  GROUP 


1.  PURPOSE:  This  is  a  brief  statement  identifying  the  weapon 
system  for  which  the  TIWG  is  being  established. 

Example:  To  formally  charter  the  *  TIWG,  comprised  of 
the  command  representatives  for  the  agencies  listed  in  para  2 
below. 

2.  MEMBERSHIP:  List  organizations  providing  members.  Include 
organizational  addresses,  office  symbols,  electronic  message 
addresses,  and  AUTOVON  telephone  numbers  to  facilitate 
communication  between  member  organizations. 

Example : 

a.  The  *  TIWG  will  be  composed  of  one  representative 
(principal)  of  each  of  the  following: 

(1)  Program  Manager/MATDEV 

(2)  Combat  Developer /Functional  Proponent. 

(3)  Developmental  Tester. 

(4)  Developmental  Independent  Evaluator /Assessor. 

(5)  Operational  Tester. 

(6)  Operational  Independent  Evaluator. 

(7)  Logistician. 

(8)  Surviveability/Lethality  Analysis  Directorate. 

(9)  Trainer. 

(10)  Threat  Integrator. 

(11)  Other  commands/ agencies /services  (when 
appropriate) . 

b.  In  addition  to  the  members  listed  above, 
representatives  of  the  agencies  listed  below  are  also 
included  in  this  TIWG.  These  members  will  attend  *  TIWG 
meetings  in  an  advisory  role  (such  as  providing  comments  on 
plans  and  reports  and  coordinating  actions  within  their _ _ 
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respective  organizations  as  appropriate  in  accordance  with 
their  assigned  mission) . 

3.  OBJECTIVE:  Specific  objective  of  each  TIWG  are  listed. 

Example:  The  objective  of  the  *  TIWG  is  to  provide  a 

forum  for  test  planning  and  integration  to  ensure  an  adequate 
and  comprehensive  test  program  to  fully  validate  the  system. 

4.  PROCEDURES:  The  procedures  section  provides  the  broad, 
general  guidelines  under  which  the  TIWG  will  operate.  The 
method  of  calling  meetings,  representation  by  members, 
developing  agenda  items,  and  conducting  meetings  are 
included.  The  organization  of  the  TIWG  is  shown  including  the 
other  subcommittees  or  working  groups  as  required.  In 
addition,  the  interface  of  the  TIWG  with  other  activities 
such  as  design  engineering,  simulation,  targets  management, 
etc. ,  is  shown.  Procedures  are  also  provided  for  handling 
open  agenda  items,  resolution  of  problems  and  preparation  of 
minutes  of  each  TIWG  meeting.  Maximum  use  should  be  made  of 
correspondence  to  resolve  issues  in  order  to  reduce  frequency 
of  meetings. 

Example: 

a.  After  coordination,  with  principal  members,  meetings 
will  convene  at  the  call  of  the  chairperson,  who  will  provide 
for  the  recording  and  distribution  of  minutes  of  meetings. 

b.  Not  less  than  2  weeks  prior  to  each  meeting,  the 
chairperson  will  provide  each  member  agency  with  notification 
of  the  time,  place,  and  agenda  for  the  proposed  meeting. 

c.  Member  agencies  will  be  responsible  for  ensuring 
their  own  representation  and  such  additional  supplementary 
representation  as  may  be  indicated  by  the  agenda. 

d.  Test  integration,  logistics,  concepts,  and  training 
subcommittees  will  be  established. 

e.  Members  will  be  responsible  for  action  items  related 
to  their  functional  areas  that  are  specified  on  an  Action 
Item  List  (AIL).  The  AIL  will  be  revised  by  the  agencies' 
representatives  at  each  meeting.  Such  additions  or  deletions 
as  recommended  by  agency  representatives  attending  will  be 
reviewed  by  the  group  and  an  updated  AIL  will  be  provided  as 
part  of  the  minutes. 


Figure  8-1.  TIWG  Charter  (Strawman) 


f.  The  TIWG  members  will  provide  inputs  and 
recommendations  with  regard  to  modification  and  revision  to 
the  TEMP. 

g.  Disagreements  on  matters  of  substance  will  be 
elevated  from  the  TIWG  to  the  next  higher  level  of  review  for 
adjudication.  Such  matters  are  brought  to  the  attention  of 
the  DUSA(OR)  for  resolution  or  guidance  if  agreement  cannot 
be  reached  at  lower  levels  of  review. 

5.  DISTRIBUTION:  This  section  includes  distribution  to  be 
made  of  the  TIWG  Charter,  changes  thereto,  minutes  of 
meetings,  plans,  reports,  etc. 

Example : 

a.  This  charter,  minutes  of  all  meetings,  and  all 
issues  of  the  *  TIWG  AIL  shall  be  distributed  to  each  *  TIWG 
principal  member  within  10  working  days  after  the  meeting. 

b.  If  the  minutes  do  not  adequately  reflect  a  member's 
understanding  of  what  was  accomplished  at  a  TIWG  meeting,  or 
if  a  member  organization's  position  changes,  this  should  be 
brought  to  the  attention  of  the  chairperson  for  correction  or 
added  as  an  action  item  to  the  next  TIWG  Agenda  within  two 
weeks  after  receipt  of  the  minutes. 

c.  Additional  supplemental  distribution  of  meeting 
minutes  and  AIL  will  be  as  recommended  by  the  group. 

d.  Copies  of  T&E  documentation,  both  government  and 
contractor,  will  be  provided  to  all  TIWG  members. 


Signature  Block 
TIWG  Chairman 
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Chapter  9 

Test  Support  Packages  (TSP) 


Section  I 
Introduction 


9-1.  General 

Test  support  packages  are  provided,  as  stated  above,  to  support 
conduct  of  Army  testing  for  new  materiel  undergoing  development 
and  fielding.  TSPs  are  primarily  used  during  developmental  and 
operational  testing  (DT  and  OT)  prior  to  milestone  III 
(production  decision) .  They  include;  System  Support  Package 
(SSP) ,  New  Equipment  Training  Test  Support  Package  (NET  TSP)  , 
Doctrinal  and  Organizational  Test  Support  Package  (D&O 
TSP) , Training  Test  Support  Package  (Training  TSP) ,  and  Threat 
Test  Support  Package  (Threat  TSP) .  TSP  for  Automated  Information 
Systems  (AIS)  will  be  tailored  as  noted.  Description  of  the  five 
TSP  follows. 

a.  System  Support  Package  (SSP).  A  set  of  support  elements 
(equipment,  manuals,  expendables,  maintenance  tools/parts) 
planned  for  a  system  in  the  operational  (deployed)  environment, 
provided  before,  and  tested  and  evaluated  during  developmental 
tests  (DT)  and  operational  tests  (OT) ,  to  determine  the  adequacy 
of  the  planned  support  capability.  The  SSP  is  provided  by  the 
PEO/PM/MATDEV. 

b.  New  Equipment  Training  Test  Support  Package  (NET  TSP) .  NET 
plan  is  prepared  by  the  PEO/PM/MATDEV  in  accordance  with  AR 
350-35  to  support  training  development  for  new  materiel  and 
systems,  including  conduct  of  test  and  evaluation  of  new 
equipment.  Based  on  the  NET  plan,  the  PEO/PM/MATDEV  prepares,  as 
appropriate,  a  NET  TSP.  The  NET  TSP  is  provided  to  the  training 
developers  and  testers.  It  is  used  to  train  player  (operators  or 
users  of  new  materiel)  personnel  for  technical  testing;  and 
conduct  training  of  instructor  and  key  personnel  who  train  player 
personnel  for  user  testing.  The  TNGDEV  uses  the  NET  TSP  to 
develop  the  training  test  support  package  (training  TSP) . 

c.  Doctrinal  and  Organization  Test  Support  Package  (D&O  TSP) . 
The  D&O  TSP  is  a  set  of  documentation  prepared  or  revised  by  the 
CBTDEV  for  each  operational  test  supporting  a  milestone  decision. 
Paragraphs  or  elements  in  the  D&O  TSP  not  needed  (as  determined 
by  CBTDEV)  will  be  annotated  as  "not  required"  in  the  D&O  TSP. 
Major  components  are: 
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(1)  Means  of  employment. 

(2)  Organization. 

(3)  Logistics  concepts. 

(4)  Operational  Mode  Summary /Mission  Profile  (OMS/MP) . 

(5)  Test  setting. 

d.  Threat  Test  Support  Package  (Threat  TSP) .  The  Threat  TSP 
is  a  document  or  set  of  documents  that  provide  a  description  of 
the  threat  that  the  new  system  will  be  tested  against. 

e.  Training  Test  Support  Package  (Training  TSP) .  The 
Training  TSP,  consists  of  materials  used  by  the  trainer  to  train 
test  players  and  by  the  tester  in  evaluating  training  on  a  new 
materiel  system.  This  includes  training  of  doctrine  and  tactics 
for  the  system  and  maintenance  on  the  system.  It  focuses  on  the 
performance  of  specific  individual  and  collective  tasks  during 
user  testing  of  a  new  system.  The  Training  TSP  is  prepared  by 
the  proponent  trainer  (e.g.  TRADOC) . 

9-2.  Applicability 

Test  support  packages  apply  to  all  programs.  This  includes  \ 

developmental,  non-developmental,  and  changes  associated  with 

system  development/acquisition  acquired  under  the  auspices  of  AR 

70-1  for  materiel  systems  and  AR  25  series  for  Automated 

Information  Systems  (AIS) .  Test  support  packages  are  required  to 

support  developmental  and  operational  testing  (DT  and  OT)  for 

both  materiel  and  information  systems. 

9-3 .  Background 

Army  reorganizations  of  test  and  evaluation  assets  during  1988 
and  1990,  and  subsequent  mission  changes  affecting  the  materiel 
acquisition  process  for  development  and  fielding  of  new,  modified 
or  product  improved  materiel,  resulted  in  major  revisions  and 
rescissions  of  Army  regulations.  This  caused  a  void  in 
"identifying  requirements"  and  "time  schedules"  associated  with 
test  support  packages.  Contained  herein  is  a  summarization  and 
consolidation  of  documents  classed  as  test  support  packages  and 
corresponding  agencies'  functions  and  minimum  requirements  to 
support  test  and  evaluation. 

9-4 .  TSP  preparation  functions 

The  program  executive  officer  (PEG) ,  program  (or  project /product) 
manager  (PM)  or  materiel  developer  (MATDEV)  for  new  materiel  is 
designated  to  prepare  and  provide  system  support  package  (SSP) 
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and  new  equipment  training  (NET)  as  stated  in  AR  70-1,  AR  350-35, 
and  AR  700-127  in  support  of  conduct  of  Army  testing  of  new 
materiel.  The  combat  developer  (CBTDEV)  and  training  developer 
(TNGDEV)  are  functionally  responsible  for  preparation  of  the 
doctrinal  and  organizational  test  support  package  (D&O  TSP)  , 
training  test  support  package  (training  TSP) ,  and  threat  test 
support  package  (Threat  TSP)  as  described  in  AR  73-1  and  AR 
381-11.  AR  25-3  and  AR  73-1  prescribe  materiel,  software, 
combat,  and  training  developers  as  well  as  functional  proponents 
TSP  responsibilities  for  AIS. 

9-5.  TSP  submission  guidelines  ,  j  , • 

Figure  9-1  summarizes  the  responsible  organizations  and  delivery 
schedule  guidelines  for  the  five  test  support  packages. 

(INSERT  FIGURE  9-1) 


Section  II 

Preparation  of  System  Support  Package  (SSP) 


9-6.  Introduction  ^ 

The  SSP  is  prepared  and  provided  by  the  PEO/PM/MATDEV  of  the  new 
equipment.  The  SSP  is  a  composite  of  support  equipment  and 
documentation  that  will  be  evaluated  during  logistic 
demonstration  and  tested  and  certified  during  technical  and  user 
tests.  These  equipment  and  documentation  include  repair  parts, 
tools,  maintenance/training  manuals,  and  consumable  supplies. 

For  AIS,  a  SSP  is  prepared  by  the  PM/MATDEV  for  hardware  and 
software  (i.e.,  systems  and  operating),  and  is  attached  as  an 
enclosure  to  the  AIS.  The  SSP,  used  to  validate  the  support 
system,  is  to  be  differentiated  from  other  logistic  support 
resources  and  services  required  for  initiating  the  test  and 
maintaining  test  continuity  (AR  700-127). 

9-7.  Policy,  responsibility,  and  content  of  SSP 
PEO/PMs/MATDEVs,  in  coordination  with  CBTDEVs,  training 
developers,  technical  and  operational  testers,  logistician,  and 
the  technical  and  operational  independent  evaluators,  develop  the 
SSP  component  list  (SSPCL) .  To  influence  test  design  plans, 
draft  descriptions  of  the  system  support  package  should  be 
provided  18  months  prior  to  start  of  an  operation  test  followed 
by  approved  descriptions  14  months  prior  to  test  start.  The 
PEO/PM/MATDEV  ensures  that  the  SSPCL  is  furnished  at  least  60 
days  before  training  test  start.  The  SSPCL  will  be  reviewed  by 
the  Integrated  Logistics  Support  Management  Team  (ILSMT)/Test 
Integration  Working  Group  (TIWG)  for  compliance.  The 
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contractor-delivered  portion  of  the  SSP  will  be  directed  by  the 
PEO/PM/MATDEV.  The  SSP  will  be  provided  to  the  test/training 
site  at  least  30  days  before  training  test  start.  The  SSP  will 
be  thoroughly  tested  and  evaluated  during  DT,  OT,  and  any 
subsequent  tests  with  critical  supportability  issues  (system 
questions  to  be  answered) . 

a.  SSP  content.  The  SSP  is  evaluated  by  the  independent 
technical  and  operational  evaluators/assessors  to  confirm  the 
adequacy  of  planned  support  for  the  materiel  systems  below-depot 
level.  The  SSP  will  be  tailored  by  the  PM/MATDEV,  in 
coordination  with  the  tester  and  evaluator /assessor ,  to  the 
supportability  issues  and  overall  level  of  materiel  system 
maturity  (at  time  of  test) .  The  SSP  will  normally  include: 

(1)  Properly  validated  publications  (operator  through 
general  support  (GS)  maintenance)  by  the  materiel  developer. 

Final  draft  publications  will  be  provided. 

(2)  A  listing  of  consumable  supplies  such  as  ammunition, 
petroleum,  oils,  and  lubricants. 

(3)  The  full  range  of  spares  and  repair  parts  that  are 
proposed  for  the  initial  support  package,  onboard  spares,  and 
basic  issue  items  (BII) .  These  spares  and  repair  parts  are 
provided  to  allow  the  evaluator  to  assess  the  accuracy  of  the 
predicted  range  and  quantity  of  these  parts  and  to  assess  the 
packaging,  handling,  storage,  transportation,  transportability, 
standardization,  and  interoperability  of  these  sets  of  parts 
(Parts  in  excess  of  the  spare  or  repair  parts,  BII,  and  onboard 
spares  may  be  provided  in  order  to  ensure  the  test  is  not 
affected  by  parts  shortages.  These  "test  continuity"  spares  will 
be  provided  separately  and  are  not  a  part  of  the  SSP) .  When 
proposed  quantities  are  not  furnished  (e.g.,  when  unit  spare  or 
repair  parts  mobility  is  not  an  issue) ,  agreement  must  be  reached 
between  tester  and  evaluator.  Representative  or  simulated 
operational  environment  resupply  times  should  be  used  in 
assessing  the  overall  readiness  implications  for  evaluation 
purposes. 

(4)  Training  Devices  (TDs)  and  instructional  modules  of 
the  system  to  be  fielded.  Devices  and  modules  should  be  as 
closely  representative  of  the  final  system  as  possible  to 
facilitate  test  and  evaluation  (see  NET  TSP) . 

(5)  Special  and  all  requirements  for  common  tools  (see  NET 

TSP)  . 
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(6)  Test,  Measurement,  and  Diagnostic  Equipment  (TMDE) . 
This  includes  hardware  and  software  that  will  be  required  and 
tested  with  the  basic  system.  When  applicable,  a  representative 
sample  of  below-depot  automatic  test  equipment  (ATE)  application 
will  be  tested  as  agreed  to  by  the  integrated  logistic  support 
management  team  (ILSMT)  and  test  integration  working  group 
(TIWG) .  Before  system  deployment,  all  test  program  set  (TPS)  and 
any  alternate  troubleshooting  procedures  will  be  tested  for 
adequacy  of  their  three  principal  parts  -  software,  inter¬ 
connecting  device,  and  instruction  manual.  TPS  will  be  fielded 
with  the  supported  system;  therefore,  a  deficiency  in  the  TPS  for 
the  materiel  system  will  be  treated  the  same  as  a  hardware 
deficiency. 

(7)  Proposed  type  of  MOS  and  skill  level. 

(8)  Transportation  and  materiel  handling  equipment. 

(9)  Calibration  procedures  and  equipment. 

(10)  Mobile  support  facilities  (for  example,  shop  vans, 
supply  vans,  and  calibration  vans) . 

(11)  Other  support  equipment  (such  as  generators,  trucks, 
trailers,  and  environmental  equipment) .  All  equipment  necessary 
to  form  the  operational  configuration  of  the  system  will  be 
included. 

b.  Waivers  SSP  requirements  for  Developmental  Tests  (DT)  and 
Operational  Tests  (OT) .  Evaluation  of  materiel  system 
supportability  is  mandatory  (See  AR  73—1)  during  test  and 
evaluation  for  materiel  in  the  acquisition  process. DT,  OT 
(Government-conducted,  system-level  portion) ,  and  subsequent 
tests  will  not  be  initiated  when  SSP  shortages  prevent  the 
adequate  testing  of  critical  supportability  issues.  Specific 
requests  for  waiver  of  this  requirement  will  be  granted  only  for 
compelling  reasons  and  only  when  a  get-well  plan  has  been 
developed  that  verifies  adequate  testing  will  be  performed  to 
correct  support  deficiencies  before  fielding. 

(1)  Requests  for  waiver  of  SSP  requirements,  if  required, 
for  Army  acquisition  category  (ACAT)  I  and  ACAT  II  systems,  will 
be  submitted  thru  OPTEC  to  HQDA  (DALO-SM) ,  WASH  DC  20310-0547. 

(2)  For  ACAT  III  and  ACAT  IV  systems,  PEO/PM/MATDEVs  have 
authority  to  approve  request  for  waiver  of  SSP  requirements  for 
DT.  OPTEC  or  the  designated  operational  test  proponent  has 
waiver  authority  for  lOTE.  Waivers  granted  will  be  reported  to 
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HQDA  (DALO-SM) ,  WASH  DC  20310-0547. 

(3)  All  requests  for  SSP  waivers  will  be  coordinated  with 

the; 


(a)  Logistician  (AMXSY-LX;  AMSAA) . 

(b)  Participating  PEO/PM/MATDEVs. 

(c)  CBTDEVs,  trainers,  testers,  and  independent 
evaluators/assessors. 

(d)  U.S.  Army  Central  TMDE  Activity  (CTA)  when  TMDE 
items  specified  are  not  available. 

(e)  Other  appropriate  military  services  and  agencies. 

(f)  ILSMT  members. 

(4)  The  SSP  waiver  request  will  include; 

(a)  Justification  for  the  request. 

(b)  A  description  of  the  SSP  shortfalls  related  to 
specific  critical  supportability  issues. 

(c)  A  statement  of  when  the  SSP  will  be  furnished  and 
the  test  design  change  required  to  accommodate  the  SSP  shortfall. 

(d)  A  get-well  plan  to  ensure  that  comprehensive 
supportability  testing  will  be  conducted  in  sufficient  time  to 
correct  deficiencies  before  fielding  (for  interim  contractor 
supported  (ICS)  systems,  SSP  deficiencies  will  be  corrected  prior 
to  transition  to  organic  support) . 

c.  If  a  waiver  is  required  for  a  SSP,  the  request  for  waiver 
will  include  the  information  listed  in  paragraph  3-32b(4)  of  AR 
700-127,  which  is;  (1)  justification;  (2)  description  of  SSP 
shortfalls;  (3)  date  the  SSP  will  be  provided;  and  (4) 
deficiencies/correction  get-well  plan  (supportability) .  Due  to 
the  required  approval  process,  waiver  requests  must  be  submitted 
as  soon  as  an  anticipated  need  arises.  A  waiver  will  only  be 
granted  for  compelling  reasons. 

(1)  Processing  for  Acquisition  Category  (ACAT)  I  and  ACAT 
II  systems  —  the  materiel  proponent  will  coordinate  an  SSP 
waiver  request  for  both  DT  and  OT  with  appropriate  logistical 
agencies.  The  materiel  proponent  will  forward  the  fully 
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coordinated  SSP  waiver  request  through  HQ  USAMC  (AMCDE-AR)  to 
HQDA  (DALO-SM)  with  a  copy  furnished  to  HQ  USAMC  (AMCSM-SI) . 

Upon  receipt  (from  HQDA)  of  waiver  approval  or  disapproval 
correspondence  for  DT,  the  materiel  developer  will  forward  a  copy 
of  the  correspondence  to  agencies  involved,  TIWG  members,  the 
designated  test  activity  (usually  TECOM) ,  and  HQ  USAMC  (AMCDE-AR 
and  AMCSM-SI) .  For  OT,  the  correspondence  should  be  provided  to 
logistical  agencies,  OPTEC,  or  other  designated  OT  proponent,  and 
HQ  USAMC  (AMCDE-AR  and  AMCSM-SI) . 

(2)  Processing  for  ACAT  III  and  ACAT  IV  systems —  the 
materiel  developer  will  coordinate  the  waiver  request  with 
appropriate  ILSMT  and  TIWG  members.  This  coordination  should 
take  place  at  a  scheduled  ILSMT  meeting.  Upon  receipt,  the 
materiel  proponent  will  forward  a  copy  of  the  approval  or 
disapproval  correspondence  to  affected  ILSMT  members.  A  copy  of 
approved  waivers  will  also  be  provided  to  HQDA  (DALO-SM) . 

(3)  Requests  for  waiver  of  type  classified  AIS  will  be  as 
stated  above.  For  non-type  classified  AIS  the  materiel  developer 
will  prepare  the  request  for  TIWG  concurrence  and  distribution  by 
the  AIS  functional  proponent  (AR  25-3) . 

d.  It  must  be  stressed  that  the  SSP  is  tailored  to  the 
system-peculiar  requirements  and  related  to  supportability 
testing  issues.  However,  once  the  SSP  for  any  testing  phase  is 
developed  and  coordinated,  it  should  not  be  compromised. 

9-8.  Process  and  procedures 

The  SSP,  for  support  of  Army  Test  and  Evaluation  (AR  700-127, 
paragraph  3-32)  is  the  prototype  for  the  planned  system  support. 
It  is  a  composite  of  the  support  resources  that  are  required  to 
support  the  materiel  system  when  fielded.  The  SSP  will  be 
evaluated  as  part  of  the  logistics  demonstration  (LD)  during 
developmental  testing. 

a.  SSP  sufficiency.  The  PEO/PM/MATDEV,  in  coordination  with 
the  independent  evaluators,  will  ensure  that  the  SSP  is 
sufficient  to  permit  evaluation  of  logistic  supportability  issues 
in  the  test  and  evaluation  master  plan  (TEMP) .  The  SSP  does  not 
include  those  logistic  support  resources  and  services  required  by 
the  tester  to  sustain  the  continuity  of  tests  and  demonstrations 
(for  example,  test  site  facilities,  and  administrative  support 
vehicle  available  at  the  test  activity) . 

b.  Draft  SSP  Component  List  (SSPCL)  delivery.  The 
PEO/PM/MATDEV  will  ensure  a  draft  SSPCL  is  developed  in 
accordance  with  DODD  5000  (MIL-STD  1379-D,  8100  DID)  series  for 
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DT,  initial  operational  test  and  evaluation,  follow-on 
operational  test  and  evaluation,  materiel  change  test  and  any 
other  tests  with  critical  supportability  issues.  The  PM/MATDEV 
will  furnish  the  draft  SSPCL  90  days  prior  to  test  to  the 
ILSMT/TIWG  members,  who  will  review  and  identify  SSP  components 
required  for  each  test  in  sufficient  time  for  the  PEO/PM/MATDEV 
to  acquire  and  deliver  the  SSP. 

c.  Final  SSPCL  delivery.  At  least  60  days  prior  to  the 
training  test  start,  the  PEO/PM/MATDEV  will  provide  two  copies 
(or  as  otherwise  specified)  of  the  final  SSPCL  to:  any  interested 
activity;  TECOM  (AMSTE-TA-H) ;  Logistician  (AMXST-LX) ;  Operational 
Test  and  Evaluation  Command  (CSTE-OPM) ;  TRADOC  (ATCD-TRADOC 
action  office;  ATTG-YP) ;  other  combat  developers,  users,  testers, 
and  independent  evaluators  such  as  Army  Materiel  System  Analysis 
Activity  (AMXSY-LX  and  AMXSY-R) ;  the  Army  Combined  Arms  Support 
Command  (CASCOM) ;  and  U.S.  Army  Training  Support  Center  (USATSC) , 
ATTN:  ATIC-DM. 

d.  SSP  delivery.  A  complete  SSP  will  be  delivered  to  the 
test  activity  at  least  30  days  prior  to  test  training  initiation. 
When  the  SSP  includes  items  available  in  the  Army  inventory,  the 
responsible  PEO/PM/MATDEV  will  ensure  the  on-site  availability  of 
such  items.  Upon  receipt,  test  activities  will  inventory  the  SSP 
and  report  shortages  to  the  independent 

evaluator/assessor/analyst  (AMSAA,  Operational  Evaluation  Command 
(OEC) ,  or  TECOM) ,  the  logistician  (AMSAA,  AMXSY-LX) , 
PEO/PM/MATDEV,  and  MRSA  (AMXMD-E)  at  least  25  days  prior  to 
scheduled  test  training  initiation.  If  the  independent 
evaluator /assessor  determines  that  SSP  shortages  exist  which 
prevent  the  adequate  evaluation  of  any  supportability-related 
issues,  test  start  will  be  suspended  until  the  complete  SSP  is 
available,  or  a  waiver  is  obtained  by  the  materiel  proponent. 

The  Test  Incident  Reporting  system  (Capstone  volume  of  this 
pamphlet  and  AR  73-1)  will  be  used  for  reporting  the  SSP 
inventory,  where  possible  for  technical  tests  and  evaluations. 


Section  III 

Preparation  of  New  Equipment  Training  Test  Support  Package  (NET 
TSP) 


9-9.  Introduction 

The  NET  TSP  is  developed  by  the  PEO/PM/MATDEV.  The  NET  TSP 
provides  an  equipment-specific  training  program  for  the  training 
developer  or  subject  matter  expert  (instructor)  to  develop  a 
training  program  to  train  troops  who  will  be  used  in  a  specific 
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operational  or  user  test.  The  NET  TSP  contains  a  combination  of 
equipment-specific  documents,  training  aids,  training  devices, 
training  simulators,  programs  of  instruction  (POIs)  and  lesson 
plans. 


a.  NET  TSP  uses  for  tests.  The  training  developer  (TNGDEV) 
or  training  proponent  will  use  the  NET  TSP  to  develop  the 
Training  Test  Support  Package  (Training  TSP)  used  by  operational 
test  participants  in  support  of  operational  test  execution  (see 
para  8.7)  .  The  developmental  tester  will  use  it  to  conduct  all 
technical  tests  during  the  development  process. 

d.  NET  TSP  for  Automated  Information  Systems  (AIS) .  A  NET 
TSP  will  be  prepared  by  the  PM/MATDEV  for  system  hardware  and 
software  and  provided  with  the  AIS  to  the  functional  proponent 
for  support  of  the  planned  testing  assessments. 

9-10.  Planning 

The.  PEO/PM/MATDEV  will  program,  budget,  and  fund  the  preparation 
and  execution  of  the  NET  TSP  for  each  materiel  system.  This 
includes,  but  is  not  limited  to,  training  courses,  travel  and  per 
diem  for  Instructor  and  Key  Personnel  Training  (IKPT)  for 
instructor  personnel  support  in  Early  User  Test  and 
Experimentation  (EUTE) ,  force  development  test  and 
experimentation  (FDTE) ,  limited  user  test  (LUT) ,  follow-on 
operational  test  and  evaluation  (FOTE) ,  and  Initial  Operational 
Test  and  Evaluation  (lOTE) .  The  development  of  NET  TSP  should  be 
prepared  in  conjunction  with  development  of  the  system  support 
package  since  they  are  mutually  supportive  of  each. 

9-11.  Process  and  procedures 

a.  Requirements.  The  NET  TSP  will  be  prepared  by  the 
materiel  developer.  It  will  include  all  training  material 
required  to  train  operators  and  maintainers  of  system  peculiar 
tasks.  The  system  support  package  must  support  the  NET  TSP  and 
must  be  developed  in  parallel  with  the  NET  TSP.  Preparation  of 
the  NET  TSP  includes  any  contractor  developed  training  to  be 
provided  in  support  of  operational  testing. 

b.  Format  of  the  NET  TSP.  Format  and  contents  of  the  NET  TSP 
are  as  follows: 

(1)  Title  of  Materiel  System. 

(2)  Training  Aids.  (e.g.,  transparencies,  35mm  slides, 
student  handouts,  and  blackboard) . 
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(3)  POI/Lesson  Plans.  (e.g. ,  draft  or  final). 

(4)  Technical  Manuals.  (May  be  draft,  commercial  or 
other) . 

(5)  Points  of  Contact  (POC) .  (Support  Agency's  POC  name 
and  telephone  number  required  for  initial  coordination) . 

(6)  Remarks.  (Notes  reflecting  clarification  of  above  (e. 
g.,  time  schedules;  support  package  components;  additional 
support  required  to  be  placed  in  the  system  for  test 
sustainment) ) . 

(7)  Maintenance.  (All  maintenance  charts/ literature 
should  be  included) . 

c.  Schedule.  Milestones  for  providing  NET  TSP  will  be 
identified  by  the  operational  tester  in  either  the  Test 
Evaluation  Master  Plan  (TEMP)  or  the  Outline  Test  Plan  supporting 
the  Army's  Test  Schedule  and  Review  Committee  (TSARC) .  TSARC 
directed  tests  will  be  identified  18  months  prior  to  the  required 
delivery  date.  The  training  TSP  for  developmental  systems  will 
be  provided  NLT  six  months  prior  to  lOTE  training  start  date. 

For  nondevelopmental  items,  the  NET  TSP  will  be  provided  at  least 
two  months  before  lOTE  training  start.  The  NET  TSP  will  be 
provided  to  the  Training  Developer  as  a  package  after  completion 
of  Instructor  and  Key  Personnel  Training  (IKPT) .  At  a  minimum, 
IKPT  must  be  scheduled  to  be  completed  six  months  prior  (two 
months  for  nondevelopmental  items)  to  the  start  of  test  player 
training  in  support  of  an  lOTE  for  a  Milestone  Decision  Review 
III.  For  EUTE  and  LUT,  a  NET  TSP  is  required  three  months  prior 
to  test  start.  NET  TSP  should  be  provided  to  the  technical 
tester  30-60  days  prior  to  technical  test  start.  The  milestone 
for  delivery  of  the  NET  TSP  to  the  technical  tester  should  be 
shown  in  TEMP.  The  NET  TSP  will  be  planned,  developed  and 
executed  in  coordination  with  the  trainer  and  concurrently  with 
the  system  support  package.  '  The  NET  TSP  will  be  delivered  to  the 
TNGDEV  or  training  proponent  NLT  six  months  prior  to  start  of 
unit  training  (developmental  or  operational  test) .  Deliveries  of 
the  NET  TSP  should  be  met  even  though  the  PEO/PM/MATDEV  may  use 
contractor  support  to  develop  TSP. 

d.  Other. 

(1)  The  materiel  developer  (MATDEV)  or  PM/PEO  pays  for 
instructors  to  support  tester  training. 

(2)  TRADOC  has  responsibility  to  train  test  players  for 
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(3)  To  provide  the  best  training  possible  with  improved 
test  results,  the  contractor,  prior  to  a  MDR  III»  may  be  allowed 
to  train  the  TRADOC  subject  matter  experts  (instructors)  as  close 
to  the  test  start  date  as  feasible  for  knowledge  retention 
purpose.  Training  aids  to  include  vehicles,  will  be  provided  to 
TRADOC  instructors  as  early  as  possible  prior  to  the  training 
test  start  date  to  train  test  players.  For  developmental 
systems,  the  six  month  lead  time  for  contractor  training  cited  in 
paragraph  9-llc  is  applicable.  However,  for  non-developmental 
item  systems  (NDI)  with  more  compressed  milestone  schedules, 
contractor  training  for  the  TRADOC  instructors  may  occur  closer 
to  start  of  the  operational  test.  To  ensure  adequate  planning, 
the  PEO/PM/MATDEV  will  notify  the  available  agencies  as  the 
acquisition  strategy  is  developed;  (e.g.,  TNGDEV,  TESTER,  and 
CBTDEV)  and  establish  mutually  satisfactory  milestone  goals. 


Section  IV 

Preparation  of  Doctrinal  and  Organizational  Test  Support  Package 
(D&O  TSO) 


9-12.  Introduction 

The  D&O  TSP  is  provided  by  the  proponent  Combat  Developer 
(CBTDEV) .  The  D&O  TSP  is  used  to  expand,  update,  and  add 
specificity  to  the  information  in  the  Mission  Need  Statement  and 
ORD  documents  to  support  planned  tests  required  to  support  a 
scheduled  decision  review  milestone.  The  degree  of  completeness 
and  the  finality  of  the  various  components  of  the  D&O  TSP  depend 
upon  the  phase  of  system  development,  i.e.,  as  additional 
knowledge  about  the  system  and  its  combat  capability  increases, 
the  more  mature  the  D&O  TSP  becomes  (MDR  I  vs  MDR  III)  .  The  D&O 
TSP  should  be  updated  prior  to  \each  of  the  major  test/analysis 
events  during  a  system's  development.  It  is  made  up  of  seven ^ 
parts;  references,  means  of  employment,  organization,  logistics 
concepts,  operational  mode  summary/mission  profile,  test  setting, 
and  coordination. 

9-13.  Policy  guidance 

A  D&O  TSP  is  a  mandatory  requirement  (AR  73-1)  in  support  of 
materiel  or  automated  information  system  development  for  conduct 
of  an  LUT,  initial  operational  test  and  evaluation  (lOTE) ,  and  a 
follow-on  operational  test  or  system  assessment.  A  D&O  TSP  may 
be  required  in  support  of  CEP,  FDTE,  and  EUTE  (as  determined  by 
the  combat  developer  or  evaluator/assessor) ,  but  content  will 
vary  based  on  test  or  experiment  requirements.  However,  as  much 
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information  as  possible  should  be  provided  to  ensure  support  of 
test  issues  as  determined  by  the  combat  developer. 

9-14.  Functional  responsibility  and  preparation 
The  combat  developer  is  responsible  for  planning  and  development 
of  the  D&O  TSP  for  each  materiel  system  undergoing  acquisition. 
The  designated  tester (s)  should  assist  the  combat  developer  or 
functional  proponent  (for  AIS,  AR  25-1  and  25-3;  i.e.,  CASCOM,  Ft 
Lee,  VA)  in  preparing  the  proposed  "test  scenario  and  concept  of 
test  employment.” 

a.  Schedule.  The  TSP  should  be  provided  27  months  prior  to 
an  lOTE,  a  LUT,  and  FOTE  as  agreed  to  by  the  TIWG  or  between  the 
combat  developer  and  tester  (CEP,  EUTE,  FDTE,  etc.)  and  as  shown 
in  TSARC  OTP  for  operational  testers. 

b.  D&O  TSP  approval.  The  combat  developer  or  functional 
proponent  (for  AIS) ,  is  responsible  for  development  and  approval 
of  all  D&O  TSP  (e.g.,  ACAT  I,  II,  III  and  IV). 

9-15.  D&O  TSP  format,  process,  and  procedures 
A  suggested  format  for  preparation  of  a  D&O  TSP  is  shown  in 
Figure  9-2.  Additional  details  describing  each  paragraph,  as 
well  as,  procedures  and  process  follow: 

(INSERT  FIGURE  9-2) 

a.  References.  The  draft  or  approved  MNS  or  ORD  may  be 
referenced  or  attached  and  aR  other  documents  supporting  the  TSP 
appropriately  referenced. 

b.  Means  of  Employment.  (This  paragraph  describes  how  the 
system  will  be  employed  and  supported.  It  includes  or  references 
documents  which  describe  the  doctrine,  tactics,  techniques, 
logistical  concepts  and  means  of  employment  for  the  tested 
system,  including  a  statement  on  new  or  revised  versus  current 
doctrine.  The  package  will  include  sufficient  detail  to  permit 
realistic  system  employment  for  conduct  of  the  specified  type 
test  (e.g.,  LUT).  It  is  required  prior  to  each  test.  It  is  used 
to  guide  the  development  of  the  TEP,  and  to  govern  user  troop 
actions  during  test.  Also,  when  appropriate,  related  documents 
for  the  new  system  or  equipment  as  well  as  interoperable/support 
equipment  will  be  shown. 

c.  Organization.  This  element  defines  military  occupational 
specialty  (MOS)  requirements,  basis  of  issue,  unit  structure, 
organizational  concept,  operating  concept,  and  lines  of  command 
or  coordination  for  units  employing  the  tested  system.  It  is 
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used  in  test  planning  to  structure  player  units.  When  new  MOSs 
are  required,  the  specific  duties  of  each  MOS  level  must  be 
included  in  the  D&O  TSP.  AR  611-1,  Military  Occupational 
Classification  structure  Development  and  Implementation,  provides 
information  for  the  development  of  this  section  of  the  D&O  TSP. 

d.  Logistics  Concepts.  This  paragraph  describes  the  concept 
for  planned  supply,  transportation,  and  maintenance  procedures 
and  methods  for  supporting  the  proposed  or  actual  test  system 
when  fielded'.  The  concept  will; 

(1)  Describe  supply  concepts  envisioned  for  class  I 
through  X  supply  items  and  outline  procedures  for  class  IX  repair 
parts  availability  for  the  system  prescribed  load  list  (PLL) 
including  maintenance  records,  PLL  records,  requests  for  class  IX 
items,  and  level  of  maintenance. 

(2)  Describe  what  supply  and  maintenance  including  repair 
parts  and  special  tools  will  be  provided  to  support  testing. 

(3)  State  system  transportation  procedures  for  rail, 
highway,  marine  and  air  movement  with  emphasis  on  new  or  changed 
requirements . 

(4)  State  the  MOS  and  duty  title  for  each  required  level 
of  maintenance. 

(5)  Describe  special  tools  and  test  equipment  required  to 
operate  and  maintain  the  system. 

(6)  Describe  each  level  of  maintenance  responsibility 
during  the  test,  e.g.,  military  personnel,  Department  of  Army 
civilian  employees  or  contractor  personnel. 

(7)  Describe  warranty  procedures  to  be  used  to  ensure 
ma intenance  conf ormi ty . 

(8)  Include  coordination  annexes  which  lists  the  agencies 
the  logistics  concept  was  staffed  with  and  accommodation  of 
comments.  The  logistics  concept  should  be  compatible  with 
concepts,  policies,  and  system  support  stated  in  AR  700-127, 

750-1  and  TM  38-710.  This  section  of  the  D&O  TSP  excludes  the 
system  support  package  provided  by  the  materiel  developer  (para 
3-27,  AR  700-127,  but  each  should  be  compatible. 

e.  Operational  Mode  Summary/Mission  Profile  (OMS/MP) .  This 
summary  presents  a  description  of  the  anticipated  mix  of  ways  the 
new  equipment  will  carry  out  its  operational  role  (OMS) .  It 
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includes  the  operational  events  and  environment  the  equipment 
experiences  from  beginninq  to  end  of  a  specific  mission  laid  out 
in  a  time-phased  approach  (MP) .  Additional,  as  required  to 
satisfy  the  purpose  of  test,  a  set  of  operational  mission 
profiles  (attack,  defense,  etc.)  will  be  shown.  This  section  is 
prepared  by  the  combat  developer  or  AIS  functional  proponent  in 
coordination  with  the  tester,  to  support  the  operational 
requirement.  Details  that  should  also  be  included  or  discussed 
for  AIS  are  wqrkload,  environment,  mobilization,  continuity  of 
operations,  data  loss,  and  system  peculiar  events,  for  example. 
The  OMS/MP  should  be  provided  to  the  tester  NLT  27  months  prior 
to  start  of  lOTE. 

f.  Test  Setting.  Total  environment  (e.g,  tactical,  threat, 
terrain,  weather  and  logistical  support)  under  which  a  system  is 
examined.  The  test  setting  defines  the  interactions  among 
threat,  friendly  actions,  and  the  environment  (or  some  specific 
geographic  location)  and  establishes  a  scenario  that  subjects  the 
system  under  test  in  the  context  of  its  total  environment,  to 
include  the  next  higher  level  system  or  organization.  The  test 
setting  should  be  compatible  with  the  threat  support  package  and 
supporting  standard  TRADOC  "war"  scenario.  Also,  the  size  of 
unit,  opposing  force,  equivalent  scale  of  operations  (e.g., 
company,  BN,  etc.)  should  be  stated. 

g.  Coordination.  This  paragraph  indicates  the  organizations 
that  normally  should  be  provided  the  D&O  TSP  for  review  and 
comment.  The  final  D&O  TSP  should  contain  an  enclosure  or 
appendix  which  details  the  results  of  the  coordination  (See 
Figure  9-3  for  suggested  TSP  coordination) .  The  combat  developer 
will  establish  appropriate  coordination  requirements  and  all 
coordination  schedules  to  support  timely  completion  of  the  D&O 
TSP  prior  to  approval.  Information  contained  in  the  D&O  TSP 
already  approved  (e.g.,  TRADOC)  shall  be  annotated  to  avoid 
inappropriate  comments. 

(INSERT  FIGURE  9-3) 

9-16.  D&O  TSP  checklist 

A  checklist  is  at  Figure  9-4  for  use  by  the  preparer  of  the  D&O 
TSP  to  ensure  that  basic  contents  of  the  TSP  are  addressed. 

(INSERT  FIGURE  9-4) 


Section  V 

Preparation  of  Threat  Test  Support  Package  (Threat  TSP) 
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9-17.  Introduction 

Proponent  combat  and  materiel  developers  are  responsible  for 
providing  threat  support,  including  validation,  to  Army  testing 
of  new  materiel  and  systems  lAW  AR  381-11  (includes  AIS  if  type 
classified  under  AR  70-1).  Threat  support  will  be  provided  in 
four  ways  by  the  proponent  threat  support  office;  (1) 
participation  in  test  planning,  (2)  preparation  of  the  Threat 
TSP,  (3)  providing  training  required  by  units  portraying  threat 
forces,  and  (4)  on-site  monitoring  the  threat  portrayal  prior  to 
and  during  the  test.  This  may  include  developmental  tests,  early 
user  test  and  experimentation  (EUTE) ,  force  development  test  and 
experimentation  (FDTE) ,  limited  user  test  (LUT) ,  initial 
operational  test  and  evaluation  (lOTE) ,  follow-on  operational 
test  and  evaluation  (FOTE) ,  and  other  tests  conducted  in  an 
operational  setting.  For  nonmajor  and  nontype  classified  AIS 
systems,  the  functional  proponent  prescribes  the  content  and 
format  of  Threat  TSP  to  support  a  test,  coordinates  its 
production,  and  approves  it  for  accuracy/adequacy  as  well  as 
monitoring  the  portrayal  of  threat  in  testing  (AR  25  series) . 

a.  Requirement.  Threat  TSP  will  be  prepared  for 
developmental  and  operational  testing  of  ACAT  I,  ACAT  II,  and 
Operational  Test  and  Evaluation  (OT&E)  systems,  when  an 
operational  threat  is  required.  Specific  testing  requirements 
for  a  given  system  will  be  determined  by  the  Test  Integration 
Working  Group  (TIWG) .  Determination  of  the  requirement  for  an 
operationally  realistic  portrayal  will  be  made  by  the  TIWG  upon 
the  recommendation  of  evaluation  organization  based  on  the 
requirements  of  the  TEMP,  COIC/AOIC. 

b.  Initial  Threat  TSP.  The  initial  Threat  TSP  (minus  test 
specifics  annexes)  is  developed  after  a  milestone  0  decision,  by 
the  combat  developer  or  functional  proponent  threat  support 
organization  to  support  future  testing  for  a  specific  system  or 
concept.  This  Threat  TSP  is  derived  from  the  system  threat 
assessment  report/ system  threat  assessment  (STAR/STA) .  The 
initial  threat  TSP  is  more  detailed  than  the  STAR/STA  and 
provides  the  threat  scenarios  to  support  a  specific  test  and 
assesses  the  impacts  of  threat-related  test  limitations.  To 
support  technical  test  requirements,  the  PEO/PM/MATDEV  proponent 
(threat  support  organization/office)  is  responsible  to  expand  and 
tailor  the  initial  Threat  TSP  for  each  test  in  which  threat  force 
operations  are  to  be  portrayed  realistically.  For  operational 
tests,  the  combat  developer  or  functional  proponent  threat 
support  activity  will  expand  and  tailor  the  initial  Threat  TSP 
for  each  operational  test  requiring  a  realistic  threat  portrayal. 
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c.  Final  Threat  TSP.  The  final  Threat  TSP  includes  an  update 
of  the  initial  Threat  TSP  plus  a  section  of  several  appendices 
that  are  developed  on  an  iterative  basis  to  support  specific 
tests  approved  by  the  TEMP.  The  appendices  become  part  of  the 
Threat  TSP  and  must  be  completed  before  final  TSP  approval  can  be 
granted . 

d.  DA  Threat  Integration  Staff  Officer  (TISO)  involvement. 

As  a  member  of  the  TIWG  for  ACAT  I,  II,  and  OSD  OT&E  oversight 
systems,  the  TISO  advises  threat  representatives  from  the  combat 
and  materiel  developers  of  tests  scheduled  and  the  anticipated 
threat  support  requirements  at  the  initial  threat  coordinating 
group  (TCG)  meeting.  TRADOC  Threat  Managers  and  AMC  Foreign 
Intelligence  Officers  serve  as  the  principal  threat  integrators 
for  operational  and  developmental  tests,  respectively. 

e.  ACAT  III  and  IV  systems  applicability.  Threat  TSPs  for 
ACAT  III  and  IV  systems  will  be  prepared  when  a  portrayal  of  a 
threat  is  required  and  determined  by  the  TIWG. 

f.  Validity.  When  approved  by  the  appropriate  agencies 
(e.g.,  TRADOC  for  user  tests),  the  Threat  TSP  describes  the 
threat  to  be  used  for  planning  and  developing  the  test  and 
portrayed  during  test  execution.  An  approved  Threat  TSP, 
however,  does  not  ensure  that  test  threat  portrayal  is  valid. 

Two  separate  approval/validation  actions  are  required  one  for 
the  Threat  TSP  and  one  for  the  threat  portrayal  during  the  test. 
The  approved  threat  is  included  in  the  approved  Test  and 
Evaluation  plan  prior  to  execution  of  test. 

g.  AIS  content.  AIS  Threat  TSP  contents  and  format  will  be 
specified  by  the  combat  developer  or  functional  proponent  for 
acquiring  information  systems,  see  para  9-17a. 

9-18.  Threat  TSP  structure  and  format 

The  Threat  TSP  will  contain  the  four  major  sections  listed  below. 
Preparation  of  the  initial  Threat  TSP  (Sections  I-III)  focuses  on 
Section  III  (technical,  tactical,  and  organizational  threat) 
since  much  of  the  information  to  complete  the  remaining  sections 
often  is  not  available  initially.  Section  IV  focuses  on  the  test 
scenario  and  is  finalized  in  coordination  with  the  tester. 

Figure  9-5  provides  a  recommended  Threat  TSP  preliminary  package 
format  for  use  as  a  guide  during  Threat  TSP  preparation.  A 
discussion  of  this  format  follows: 

(INSERT  FIGURE  9-5) 

a.  Title  page.  Self  explanatory. 
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b.  Tables  of  contents  and  illustrations.  Self  explanatory. 

c.  Section  I  Background  Information.  Self  explanatory. 

d.  Section  II  Critical  Operational  Issues  and  Criteria/ 
Additional  Operational  Issues  and  Criteria  (COIC/AOIC) .  COIC  are 
key  issues,  with  associated  scope,  criteria,  and  rationale  which 
must  be  answered  for  the  Full  Production  decision  (normally 
Milestone  III  decision  review) .  They  provide  broad  insight  into 
threat  support  requirements.  AOIC  are  developed  by  the  evaluator 
to  complement  or  support  evaluation  of  the  COIC  as  well  as 
provide  for  comprehensive  evaluation  of  the  total  system. 

Approved  COIC/AOIC  sometimes  are  not  available  when  beginning  the 
development  of  the  draft  Threat  TSP.  Draft  COIC  can  be  obtained 
from  the  combat  developer,  or  the  evaluator,  and  used  to  initiate 
preparation  of  the  Threat  TSP.  The  Threat  TSP,  however,  cannot 
be  completed  until  the  COIC/AOIC  are  approved,  nor  forwarded  for 
approval/validation  unless  the  approved  COIC/AOIC  are  included  in 
Section  II. 

e.  Section  III  Threat.  This  section  is  required 
approximately  18  months  prior  to  the  actual  test  (T-540) .  When 
initially  developed,  it  will  be  somewhat  generic  in  nature  but 
adequate  for  test  planning.  As  test  requirements  are  better 
defined,  this  section  will  be  revised  to  describe  the  specific 
threat  required  for  the  test.  Should  an  extensive  amount  of 
material  be  required,  systems  and  tactical  considerations  are  to 
be  summarized  with  references  to  more  detailed  and  approved 
intelligence  documents.  This  section  will  include; 

(1)  Specific  systems  and  units/organizations  that  are  a 
threat  to,  or  a  target  of,  the  system,  organization  .or  concept 
being  tested;  include  technical  descriptions  of  threat  systems 
and  TO&E  for  units. 

(2)  Threat  tactics,  doctrine,  techniques,  procedures  and 
flight  profiles  (as  appropriate) . 

(3)  Threat  countermeasures.  Primary  sources  of 
information  include  the  STAR  and  other  DIA-approved  intelligence 
documents. 

f.  Section  IV  Test  Specific  Appendices.  The  following 
appendices  are  essential  elements  of  the  Threat  TSP  and  will  be 
added  as  completed.  All  required  appendices  must  be  included 
with  the  Threat  TSP  when  it  is  forwarded  for  final  approval. 
Resources  identified  to  support  the  test  must  also  be  shown  in 
the  outline  test  plan  (OTP) . 
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(1)  Appendix  A  Test  Concept  (Draft  of  TEP  Chapter  3-1) . 

The  test  concept  is  developed  by  the  tester  from  Chapters  1  and  2 
of  the  TEP  prepared  by  the  evaluator.  It  will  describe  in 
detail,  the  test  scope  and  criteria.  The  test  concept  will  be 
used  to  define  the  required  threat  for  a  specific  test. 

(2)  Appendix  B  Scenario.  The  test  scenario  describes  how 
the  test  unit  and  OPFOR  operations  should  be  conducted. 

Selection  of  the  scenario  is  the  responsibility  of  the  combat 
developer  in  coordination  with  the  test  organization.  The  test 
organization,  in  coordination  with  the  threat  support  office,  is 
responsible  for  integrating  the  approved  threat  into  the 
scenario.  Normally,  test  scenarios  are  based  on  TRADOC-approved 
low-resolution  or  high-resolution  scenarios  or  other  recent  and 
related  combat  developments  actions.  All  aspects  of  the  scenario 
must  be  reviewed  from  the  threat  perspective  to  ensure  adequate 
portrayal  in  support  of  the  stated  operational  issues  and 
criteria  (QIC).  Areas  to  consider  include:  scheme  of  maneuver, 
TO&E  organization,  types  of  equipment,  tactics  and  supporting 
fires  or  forces. 

(3)  Appendix  C  Description  of  trials/ test  runs/ 
vignettes.  This  appendix  describes  how  the  test  operations  will 
be  conducted.  The  Threat  TSP  must  include  a  description  of  the 
threat  forces  and  operations  that  will  be  used  to  portray  the 
scenario  during  the  test.  Templates  showing  threat  force 
locations,  routes  of  movement,  and  listing  of  threat  force 
organizations  and  equipment  to  be  used  in  the  test  are  required. 
Inclusion  of  this  information  allows  reviewing  agencies  to 
determine  whether  or  not  the  threat  will  be  portrayed  accurately 
to  support  the  OIC. 

(4)  Appendix  D  Firer/target  matrix.  The  test  organization 
and  the  threat  support  office  representative  will  participate  in 
the  development  of  instrumentation  and  possibly  modeling  data 
requests  (Ph  and  Pk  numbers)  developed  and  submitted  by  the 
proponent  school  or,  in  some  cases,  the  test  agency.  The  data 
requests  will  normally  be  in  the  form  of  a  firer/target  matrix 
that  is  submitted  for  approval.  When  required,  the  firer/target 
matrix  is  prepared  by  the  proponent  threat  office  and  the  test 
organization. 

(5)  Appendix  E  Targets,  threat  simulators,  and/or 
surrogate  equipment.  Most  field  testing  requires  the  use  of  U.S. 
Army,  NATO  or  contractor  equipment  to  be  used  in  lieu  of  actual 
threat  systems.  Assessments  are  to  be  limited  to  features  that 
are  applicable  to  the  specific  test.  For  example,  if  the  test 
threat  systems  are  to  be  immobile  during  a  test,  then  it  is  not 
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appropriate  to  point  out  that  the  surrogate  systems  are  not  as 
fast  as  the  actual  threat  system  it  is  attempting  to  portray. 
Validation  and  accreditation  reports  contain  information  on  the 
fidelity  and  suitability  of  targets  and  surrogates,  and  threat 
simulators  used  for  testing.  These  are  listed  in  the  outline 
test  plan  and  contain  technical  information  needed  to  assess  the 
impact  of  shortcomings  related  to  the  threat  equipment  as 
potential  test  limitations.  A  list  of  all  threat-related 
equipment  required  for  the  test,  available  from  the  Outline  Test 
Plan  written  by  the  tester,  should  be  included  in  the  Threat  TSP. 
The  evaluator/tester  will  identify  the  OPFOR  equipment  required 
for  test. 

(6)  Appendix  F  Limitations.  The  test  evaluator  and  test 
organization  will  make  known  overall  threat-related  test 
limitations,  i.e.,  tactics,  equipment  or  other  aspects  which 
might  affect  the  test,  to  include  appropriate  rationale.  The 
preparer  is  required  to  assess  and  describe  the  effects  of  these 
stated  limitations,  plus  any  limitations  they  perceive,  on  the 
ability  of  the  test  agency  to  portray  a  valid  threat. 

(7)  Appendix  G  Threat  force  training  plan.  A  threat  force 
training  plan  is  mandatory  for  force-on-force  tests  or  tests 
involving  any  threat  replication  requiring  threat  player 
personnel.  The  proponent  will  develop  a  threat  force  training 
plan  to  train  the  designated  player  units  in  the  threat  tactics 
and  situations  to  be  portrayed  in  the  test.  The  threat  force 
training  plan  will  include  a  POI.  The  POI  should  be  based  on 
what  the  test  threat  force  needs  to  know  for  the  specific  test. 

9-19.  Process  and  procedures 

a.  Basis.  The  Threat  TSP  is  derived  from  the  System  Threat 
Assessment  Report/ System  Threat  Assessments  (STAR/STA) ,  which  is 
used  to  define  the  tactical  context  for  the  test.  The  Threat  TSP 
helps  define  the  operational  test  environment  described  in 
Chapter  3  of  the  test  evaluation  plan  (TEP)  and  provides  the 
threat  scenarios  for  each  test.  The  Threat  TSP  must  be 
sufficiently  detailed  to  enable  the  tester  to  accurately  portray 
the  threat  to  the  test  system  level  in  an  operationally  realistic 
test  environment  at  a  specified  post-IOC  date  (e.g.,  one  year 
after  fielding  of  the  new  equipment) .  Determination  of  the  post- 
IOC  date  and  scenario  selection  will  be  made  by  the  TIWG  based  on 
recommendations  of  the  systems  proponent  and  evaluation 
organization. 

b.  Schedule.  The  Threat  TSP  will  be  prepared/updated  to  meet 
the  testing  milestones  as  set  by  the  appropriate  TIWG.  Normally, 
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the  draft  Threat  TSP  must  be  submitted  18  months  before  the  test 
(T-540)  for  approval/validation  by  the  appropriate  authorities 
(See  Table  2-1,  AR  381-11)  which  must  be  accomplished  14  months 
before  the  test  (T-420) . 

c.  Preparation.  For  materiel  systems  or  applicable  AIS 
(under  AR  70-1) ,  the  combat  developer  or  functional  proponent 
threat  support  organization/activity  will  prepare  the  initial 
Threat  TSP,  after  a  milestone  0  decision  review,  and  all 
subsequent  iterations  that  support  operational  testing  (EUTE, 

LUT,  lOT&E,  and  FOTE)  requiring  threat  portrayal.  The 
PM/materiel  developer  proponent  threat  support  office  will  assist 
in  the  preparation  of  initial  Threat  TSP  and  prepare  all 
subsequent  iterations  that  support  developmental  testing.  Timely 
completion  of  the  Threat  TSP  depends  on  early  and  continuing 
coordination  and  the  exchange  of  test  planning  data  between  the 
test  agency  and  the  threat  support  activity. 

d.  Approval.  Guidelines  for  Threat  TSP  approval  follow: 

(1)  DA  ODCSINT  will  approve  all  Threat  TSPs  developed  for 
testing  of  ACAT  I,  II  and  OSD  OT&E  oversight  systems.  DA  ODCSINT 
will  forward  a  copy  of  the  threat  TSP  for  ACAT  ID  and  IC  systems 
to  DIA  for  review  and  comment  (through  MDR  I) . 

(2)  TRADOC,  or  the  appropriate  MACOM  responsible  for  non- 
TRADOC  managed  systems,  will  approve  all  Threat  TSPs  for  all 
system  categories  and  validate  all  Threat  TSP  developed  for 
operational  testing  of  ACAT  III  and  IV  systems,  exempted  from  OSD 
OT&E  oversight. 

(3)  AMC,  or  the  appropriate  materiel  developer,  will 
approve  Threat  TSPs  developed  for  developmental  testing  of  ACAT 
III  and  IV  systems  exempted  from  OSD  T&E  oversight. 

(4)  Approval  of  the  Threat  TSP  will  take  place  only  after 
all  sections/appendices  are  completed.  As  sections/appendices 
are  developed,  they  should  be  coordinated  with  the  MACOM  threat 
approval  authority  and  if  an  ACAT  I,  11  or  OSD  T&E  oversight 
systems,  DA  ODCSINT  for  initial  review.  This  allows  use  of  these 
sections/appendices  in  the  test  planning  process  prior  to  final 
approval  of  the  Threat  TSP.  The  Test  and  Evaluation  Plan  (TEP) 
cannot  be  finalized  prior  to  receipt  of  an  approved  Threat  TSP. 

(5)  For  each  operational  test  for  which  a  Threat  TSP  has 
been  prepared,  validation  of  both  the  Threat  TSP,  threat  force 
training  plan  (where  applicable) ,  and  the  approved  test  threat 
portrayal  is  required.  This  validation  provided  by  the  portrayal 
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validation  authority,  is  documented  in  the  Operational  Test 
Readiness  Statement  prepared  by  the  combat  developer  for 
Operational  Test  Readiness  Reviews.  Pretest  threat  portrayal 
validations  will  not  be  granted  without  completions  of 
accreditation  of  the  threat  equipment.  Threat  portrayal  is 
monitored  by  the  appropriate  threat  support  organization  and 
threat  validation  authority. 

(6)  For  AIS,  the  appropriate  functional  proponent  will 
approve  and  validate  the  Threat  TSP  and  ensure  the  accuracy/ 
adequacy  of  the  portrayal  of  threat  in  testing. 

e.  Classification.  Threat  TSP  classification  will  be  limited 
to  SECRET.  Higher  level  supplements  may  be  added  as  needed. 


Section  VI 

Preparation  of  Training  Test  Support  Package  (Training  TSP) 


9-20.  Introduction 

The  Training  TSP  is  provided  to  the  test  agency  by  the  training 
developers  (proponent)  of  the  new  system.  A  Training  TSP  is 
assembled  by  the  proponent  training  developer  for  each  affected 
operator  and  maintainer  MOS.  Where  there  are  system  cross 
proponent  responsibilities,  the  proponent  for  the  requirement 
will  be  responsible  for  assembly  of  training  materials  for 
supporting  MOS.  The  lead  proponent  will  consolidate  the  package 
and  assure  it  does  not  contain  conflicting  requirements.  The 
Training  TSP  contains  information  used  by  the  trainer  to  train 
test  players  and  for  the  tester's  use  in  evaluating  training  on  a 
new  materiel  system.  It  focuses  on  the  performance  of  specific 
individual  and  collective  tasks  during  user  testing  of  a  system. 
The  Training  TSP  package  should  be  updated  prior  to  each  of  the 
major  user  test/analysis  events  (early  user  test  and 
experimentation,  LUT,  follow-on  operational  test  and  evaluation, 
and  initial  test  and  evaluation)  during  a  system's  development  or 
as  required  by  the  TEMP  or  OTP.  Training  TSP  for  AIS  will  be 
tailored  to  the  skills  and  abilities  of  the  target  audience 
scheduled  to  use  the  system.  If  there  are  no  specified  MOS  to 
use  the  AIS,  it  should  be  addressed  and  the  users  described. 

a.  Training  TSP  contents.  A  Training  TSP  consists  of  the 
following:  The  System  Training  Plan  (STRAP) ;  the  Training 
Certification  Plan:  the  training  schedule,  the  program  of 
instruction  (POI)  for  each  military  occupational 
specialties/special  skill  identifiers  (MOS/SSI)  involved  in  the 
testing;  training  data  requirements;  the  Army  Training  Evaluation 
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Program  (ARTEP) /Mission  Training  Plan  (MTP)  and  ARTEP/MTP 
changes;  a  list  of  training  aids,  training  devices,  and  embedded 
training  components;  a  target  audience  description;  soldier 
training  publications  or  changes,  crew  drills,  lesson  plans; 
ammunition,  targets,  and  ranges  required  for  training;  and  a 
critical  task  list. 

b.  Training  TSP  input.  Logistics  oriented  schools  and 
non-proponent,  schools  which  manage  MOSs  involved  with  the  new 
system  are  responsible  for  developing  Training  TSP  input  (e.g., 
POI;  Lesson  plans;  STRAP  changes;  training  data  requirements; 
ARTEP/MTP  changes;  target  audience  descriptions;  crew  drills; 
ammunition;  targets  and  ranges  required  for  training;  and 
critical  task  list)  to  the  lead  proponent.  This  is  in  addition 
to  the  NET  TSP  provided  by  the  materiel  developer.  All  Training 
TSP  input  must  be  provided  in  sufficient  time  from  responsible 
agencies  to  the  training  developer  to  allow  the  TSP  to  be 
submitted  on  time  to  the  tester  (e.g.,  60  days  prior  to  start  of 
test  player  training) . 

c.  AIS  Training  TSP.  When  required,  the  AIS  Training  TSP 
will  be  prepared  as  specified  by  the  training  proponent  for  the 
information  system  under  development. 

d.  Support  Documentation.  The  Training  TSP  may  provide  or 
make  reference  to  supporting  documentation  to  the  TSP. 

Attachment (s)  should  depend  on  availability  of  referenced 
documents . 

9-21.  Policy,  process,  and  procedures 

The  proponent  training  developer  (or  when  TRADOC  is  the 
proponent)  is  responsible  for  developing,  coordinating,  and 
providing  the  Training  TSP  to  the  test  agency.  Training  TSP 
items  identified  in  paragraph  9-2 Id  below  will  be  submitted  for 
approval  to  HQ  TRADOC  or  Major  Army  Commands  (MACOMs)  assigned 
responsible  for  non-TRADOC  systems. 

a.  Initial  Submission.  The  initial  Training  TSP  consists  of 
the  approved  STRAP/ training  data  requirements,  and  the 
Certification  Plan.  It  provides  the  test  agency  with  the 
training  concept  for  the  system,  the  training  issues  upon  which 
the  trainer  requires  data,  and  the  method  for  training  test 
players  are  trained.  The  initial  submission  is  due  to  the  test 
agency  from  Test  (T)  start  minus  (-)  T-18  months,  or  as  specified 
in  the  OTP. 

b.  Final  Submission.  The  Training  TSP  is  a  full  package.  It 
is  prepared  following  instructor  and  key  personnel  training 
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(IKPT)  and  receipt  of  the  New  Equipment  Training  Test  Support 
Package  which  should  occur  six  months  prior  to  start  training 
date  for  developmental  systems  and  not  later  than  two  months 
prior  to  start  training  date  for  NDI  systems.  The  final  Training 
TSP  is  submitted  to  HQ  TRADOC  60  days  (TRADOC  systems)  prior  to 
the  commencement  of  test  player  training. 

c.  Functions. 

(1)  HQ  TRADOC,  ATTG-Y  (TRADOC  systems  only;  other  TNGDEV 
should  confirm  procedures  with  responsible  MACOM) . 

(a)  Provides  guidance  on  preparation,  coordination, 
approval  and  distribution  of  the  Training  TSP. 

(b)  Serves  as  approving  authority  for  all  STRAPS  and 
Training  TSPs. 

(c)  Serves  as  the  TRADOC  policy  element  for  the  STRAP 
and  the  Training  TSP. 

(2)  USAOPTEC. 

(a)  Reviews  draft  Training  TSP  and  provides  comments  to 
proponents . 

(b)  Ensures  Training  TSP  is  included  as  part  of  the 
Test  and  Evaluation  Plan  (TEP)  development  process. 

(c)  OPTEC  will  host  the  operational  readiness  review 
(OTRR) .  OPTEC  will  ensure  all  training  is  completed  prior  to 
start  of  test  (two  week  minimum) . 

(3)  Training  Proponent  (e.g.  TRADOC  schools  for  TRADOC 
Systems) . 


(a)  Prepares  initial  and  final  Training  TSPs  in 
coordination  with  supporting  schools. 

(b)  Forwards  approved  copies  of  initial  and  final 
Training  TSPs  to  the  tester. 

d.  Training  TSP  content. 

(1)  Initial  submission.  The  Initial  Submission  Training 
TSP  contains: 

(a)  System  Training  Plan  (STRAP) .  The  STRAP  should  be 
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approved  prior  to  including  it  in  the  Training  TSP.  Approval  of 
the  Training  TSP  should  not  be  construed  as  approval  of  the 
STRAP.  See  TRADOC  Reg  351-9  for  a  detailed  description  of  the 
contents  of  each  paragraph  in  the  STRAP. 

(b)  Test  training  certification  plan.  The  plan 
outlines  and  describes  the  method  and  procedures  for  evaluating 
and  certifying  individual  and  collective  pre-test  training. 
Specifically,  it  describes  the  who,  where,  and  how  training  is 
certified. 

(c)  Training  data  requirements.  Data  requirements  and 
milestones  should  be  identified. 

(2)  Final  submission.  The  submission  of  the  final 
Training  TSP  contains: 

(a)  Training  schedule. 

(b)  POI  for  each  MOS/SSI  affected. 

(c)  ARTEP/MTP  or  changes  to  ARTEP/MTP. 

(d)  List  of  training  devices,  embedded  training 
components,  and  simulators. 

(e)  Target  audience  description. 

(f)  Soldier  training  publications  or  changes. 

(g)  Crew  drills. 

(h)  Lesson  plans. 

(i)  Ammunition,  targets,  and  ranges  required  for 
training.  (Submit  to  TRADOC  for  approval.) 

(j)  Critical  MOS  task  list. 

9-22.  Checklist 

Figure  9-6  provides  a  checklist  for  use  during  preparation  of  the 
Training  TSP. 


(INSERT  FIGURE  9-6) 
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1.  Title  Page  (type  of  test,  system,  date,  etc.)* 

2.  References. 

3.  Means  of  Employment. 

a.  Field  Manuals  (FMs) . 

b.  Field  Circulars  (FCs) . 

c.  Training  Circulars  (TCS) . 

d.  Soldiers  Manuals  (SMs) . 

e.  Operators  Manuals. 

f .  Tactical  Unit  Standing  Operating  Procedures  (TAC  SOP) . 

g.  Communications-Electronic  Operating  Instructions 

(CEOI)  . 

h.  Equipment  Storage  Plans  (Load  lists) . 

4.  Organization. 

a.  Basis  of  Issue  Plan  (BOIP) . 

b.  Qualitative  and  Quantitative  Personnel  Requirements 
Information  (QQPRI) . 

c.  Organization  Plan. 

(1)  Introduction. 

(2)  System  Description. 

(3)  Organizational  Concept  (Unit). 

(4)  Operating  Procedures. 

5.  Logistics  Concept. 

a.  Purpose. 

b.  Source. 

c.  Description. 

d.  Supply. 

e.  Transportation. 

f.  Maintenance. 

g.  MOS  by  level  of  maintenance. 

h.  Special  tools  and  test  equipment. 

i.  Coordination  Annex. 

6.  Mission  Profiles. 

7.  Test  Setting. 

8.  Coordination. 

9.  Approval  authority. 

NOTE:  Content  will  vary  based  on  purpose  of  TSP  and/or  MDR 

decision  and  as  deemed  necessary  by  the  CBTDEV  and  Evaluator 
(tester) . 

FIGURE  9-2  RECOMMENDED  FORMAT  FOR  A  DOCTRINAL  AND 

ORGANIZATIONAL  TSP 


ORGANIZATION 


TYPE  TEST/ EXPERIMENT 
♦EUT&E/OT&E  FDT&E/FDEV/CEP  TEST 


USAOPTEC/TEXCOM  X 

CBTDEV  (PROPONENT)  X 

CO-PROPONENT  (IF  USED)  X 

MATERIEL  DEVELOPER  info 

TECHNICAL  TEST  ACTIVITY  info 

TRADOC  SYSTEM  MANAGER  X 

TECHNICAL  EVALUATOR  info 

TRAINER  (HQ  TRADOC  or  X 

responsible  MACOM  for 
non-TRADOC  systems) 


♦NOTE:  OT&E  inclu:^■'S  LUT,  lOT  and  FOT. 


X 

X 


X 


FIGURE  9-3  TYPICAL  COORDINATION  FOR 
A  DOCTRINAL  &  ORGANIZATIONAL  TSP 


CHECKLIST  FOR  DOCTRINAL  AND  ORGANIZATIONAL 
TEST  SUPPORT  PACKAGE  (D&O  TSP) 

1.  Following  is  a  list  of  items  to  consider  during  preparation  and 
review  of  D&O  TSP: 

a.  References  and  title  page.  Administrative  information  and 
ORD/TSARC  references  (current  and  available) . 

b.  Means  of  Employment. 

(1)  Does  the  D&O  TSP  provide  a  complete,  current  listing 
of  the  doctrinal  materiel  that  will  be  required  for  the  new  system 
at  the  unit  level,  e.g.,  FMs,  FCs,  TCs,  SMs,  operators  manuals  (may 
be  included  in  the  SSP) ,  TAC  SOPs,  CEOIs,  and  load  plans? 

(2)  Does  the  D&O  TSP  provide  a  listing  of  the  doctrinal 
material  used  at  staff  levels  above  the  operating  unit  that  must  be 
changed  or  augmented  to  support  fielding  of  the  system? 
Interoperability? 

(3)  Are  drafts  of,  or  changes  to  the  listed  or  referenced 
documents  included  in  the  D&O  TSP? 

(4)  Is  the  draft  documentation  such  that  it  addresses 
system  employment  and  permits  development  of  the  TEP,  DTP  and  other 
T&E  planning  documents,  e.g.,  TEMP,  COIC,  etc.? 

(5)  Are  dates  for  delivery  of  the  Tactical  SOP, 
communication/electronic  and  loading  instructions  and  plans 
established? 

(6)  Does  the  scope  "state  the  tactical  scenario? 

c.  Organization . 

(1)  Are  draft  or  final  TOEs  for  units  employing  the  system 
up  to  battalion  level  or  ,  equivalent  included?  BOIP,  QQPRI 
referenced? 

(2)  Does  the  D&O  TSP  include  a  detailed  description  of  the 
operational  concept  for  employing  the  system  in  combat  to  include 
lines  of  communication  and  coordination  through  division  level? 

(3)  Does  the  D&O  TSP  describe  each  of  the  system 
employment  options,  e.g.,  direct  support,  general  support, 
attachment ,  etc . ? 
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(4)  Are  the  operating  procedures  for  each  option  described 
in  detail? 

(5)  Are  the  lines  of  C3  for  the  system  clearly  delineated? 

(6)  Are  the  degraded  inode(s)  of  operation  described  in 

detail? 

(7)  Are  the  various  communications  options  (wire,  radio 
(voice,  digital  data,  secure,  etc.)/  facsimile,  etc.)  described? 

(8)  Are  related  operational  and  organizational  concepts 
included  in  the  D&O  TSP?  This  applies  when  the  system  under 
development/ test  is  used  in  conjunction  with  or  to  employ  other 
systems.  An  example  of  a  system  requiring  special  treatment  is  the 
Fire  Support  Team  Vehicle  (FISTV)  which  in  addition  to  its  usual 
field  artillery  missions  may  be  required  to  employ  Hellfire 
missiles,  U.S.  Air  Force  laser  guided  and  conventional  weapons  and 
other  systems.  The  D&O  TSP  should  include  the  employment  concepts 
for  each  such  related  system. 

(9)  MOS's  discussed? 

d.  Logistics  Concepts. 

(1)  Are  the  logistical  concepts  for  the  system  through  the 
direct  support  level  incorporated  into  draft  FMs  and  support 
documents? 

(2)  Are  they  shown  in  FM  (draft/final)? 

(3)  Are  the  logistical  concepts  detailed  enough  so  that 
testing  can  be  accomplished  through  the  direct  support  level? 

(4)  Are  all  major  logistical  areas  included  (e.g.,  supply, 
maintenance,  transportation) . 

(5)  Does  the  logistics  concept  include  procedures  for  use 
of  operational  readiness  floats  (ORF)? 

(6)  Type  of  support  stated  (troop,  contract)? 

(7)  Are  there  environmental  impacts (e.g. ,  manufacturing, 
supply  actions)? 

e.  Operational  Mode  Summarv/Mission  Profile. 

(1)  Has  the  OMS/MP  been  expanded  or  updated  since  the  last 
user  test  and/or  publication  of  the  ORD? 
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(2)  Does  the  OMS/MP  describe  the  events  and  frequency  of 
occurrence  and  duration  events  in  attack,  defense,  exploitation  and 
retrograde  operations?  State  alternate  missions? 

(3)  Does  the  OMS/MP  state  the  frequency  and  duration  of 
responses  to  threat  use  of  countermeasure  such  electronic  warfare 
and/or  radio  electronic  combat,  obscurants  and  NBC  weapons? 

f .  Test  Setting. 

(1)  Does  the  setting  detail  friendly  and  threat  force 
actions  down  to  the  unit  level? 

(2)  Are  the  probable  areas  of  employment  for  the  proposed 
system  stated? 

(3)  Does  the  setting  state  the  primary  areas  of  employment 
for  the  proposed  system? 

(4)  Is  the  approved  scenario  on  which  the  test  setting  is 
based  referenced?  (Include  sequence  number  and  date  of  the 
scenario) . 

(5)  Does  the  setting  state  or  relate  to  standard  TRADOC 
theater-level  scenario  and/or  threat  support  package? 

(6)  Does  the  test  setting  identify  the  type  force 
structure  for  the  proposed  system? 

2.  After  finalizing  contents,  ensure  that  adequate  coordination  is 
accomplished.  A  suggested  coordination  list  is  shown  in  Figure  8-2 
for  the  D&O  TSP. 
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1.  Title  page.  (Preparing  agency,  information  cutoff  date,  U.S. 
system  project  office,  and  the  MACOM  or  DA  validation  date). 

2.  Tables  of  contents  and  illustrations. 

3.  Section  I  Background  Information. 

a.  Description  of  system,  organization  or  concept  to  be  tested. 

b.  Type  of  test. 

c.  Evaluating  agency. 

d.  Test  organization. 

e.  TRADOC  proponent  school. 

f.  Test  dates. 

g.  Test  location. 

h.  Simulated  location  (e.g.,  central  Europe). 

i.  IOC  of  system  being  tested. 

j.  Threat  year. 

4.  Section  II  Critical  Operational  Issues  and  Criteria/Additional 
Operational  Issues  and  Criteria  (COIC/AOIC) . 

5.  Section  III  Threat. 

a.  Specific  threat  systems  and  units/organizations. 

b.  Threat  tactics,  doctrine,  techniques,  procedures  and  flight 
profiles  (as  appropriate) . 

c.  Threat  countermeasures. 

6.  Section  IV  Test  Specific  Appendices. 

a.  Appendix  A  Test  concept  (Draft  of  TEP  Chapter  3-1) . 

b.  Appendix  B  Scenario. 

c.  Appendix  C  Description  of  trials/test  runs/ vignettes . 

d.  Appendix  D  Firer/target  matrix. 

e.  Appendix  E  Targets,  threat  simulators,  and  surrogates. 

f.  Appendix  F  Limitations. 

g.  Appendix  G  Threat  force  training  plan. 
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CHECKLIST  FOR  TRAINING  TEST  SUPPORT  PACKAGE 
(Training  TSP) 

1.  Initial  Submission  of  the  Training  TSP. 

a.  Were  development  procedures  outlined  in  TRADOC  Reg  351-9 
followed  for  the  STRAP? 

b.  Did  the  STRAP  address: 

(1)  The  system  description? 

(2)  Assumptions? 

(3)  The  training  concept? 

(4)  The  training  device  strategy? 

(5)  Significant  training  issues  at  risk? 

c.  Did  the  Test  Training  Certification  Plan  describe  the 
method  and  procedures  for  evaluating  and  certifying  individual  and 
collective  pre-test  training?  Specifically,  did  it  describe  the 
who,  where,  and  how  training  is  to  be  accomplished  and  the  method 
of  certification? 

d.  Were  the  STRAP  and  Test  Training  Certification  Plan 
submitted  within  the  time  frame  prescribed? 

e.  Did  the  Training  Data  Requirements  provide  training  issues 
outlining  the  need  for  data  on  individual/collective  performance, 
technical  manuals,  etc.? 

2.  Final  Submission. 

a.  Is  final  Training  TSP  submitted  to  HQ  TRADOC  at  least  60 
days  prior  to  the  test  date? 

b.  Does  the  final  Training  TSP  include: 

(1)  The  training  schedule? 

(2)  The  POI  for  each  MOS/SSI  affected? 

(3)  FMs/FM  Changes  References? 

(4)  The  ARTEP/MTP  or  changes  to  the  ARTEP/MTP? 


FIGURE  9-6  TRAINING  TSP  CHECKLIST 


Oi  ^  32. 


(5)  A  list  of  training  devices,  embedded  training 
components,  and  simulators? 

(6)  A  target  audience  description? 

(7)  Soldier  training  publications  or  changes? 

(8)  Crew  drills? 

(9)  Lessons  Plans? 

(10)  A  list  of  ammunition,  targets,  and  ranges  required 
for  training? 

(11)  A  critical  task  list? 

c.  Does  the  Training  TSP  include  information  from  each  MOS 
proponent  school  affected? 

d.  Does  the  Training  TSP  lay  out  who  is  responsible  for 
training  those  tasks  taught  in  the  institution  and  unit? 

e.  Does  the  TRAINING  TSP  contain  all  of  the  material  needed 
to  train  test  players  on  operator  and  maintainer  tasks? 

f.  Is  field  training  necessary?  Does  it  train  operator  crews 
to  operate  the  system  to  its  desired  capability?  Is  night  training 
appropriate? 

g.  Are  tactics,  techniques,  and  procedures  (DTT)  taught  to 
test  players?  Does  it  agree  with  the  employment  described  in 
doctrinal  manuals? 

h.  Is  there  sufficient  time  built  into  the  training  schedule 
for  the  unit  to  become  proficient  with  the  system? 

i.  Will  training  devices  be  available  to  support  test 
training? 

j.  How  much  ammunition  is  required  to  support  training?  Is 
it  supportable? 

k.  Is  the  test  player  a  "typical  soldier"  in  his  career 
field? 
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Test  and  Evaluation  in  Support  of  Integrated  Logistics  Support 

(ILS) 


Section  I 
General 


10-1.  Introduction 

This  chapter  explains  for  testers  and  evaluators  how  to  plan, 
conduct,  and  report  testing  and  CE  of  system  logistics 
supportability  (figure  10-1) . 

(INSERT  FIGURE  10-1) 

a.  This  chapter  covers  the  development  of  logistics  support 
test  requirements  and  the  conduct  of  supportability  assessments 
to  ensure  that  readiness  and  supportability  objectives  are 
identified  and  achieved.  The  ILS  Manager  must  ensure  the  ILS  T&E 
objectives  are  considered  and  that  adequate  resources  are 
available  for  ILS  T&E. 

b.  The  methodologies  described  in  this  chapter  address  T&E  of 
ILS  elements  that  should  be  available  in  the  system  support 
package  from  which  the  test  support  package  is  derived  as  well  as 
logistics  support  concepts,  doctrine,  and  organization  (see 
glossary  for  definition)  as  described  in  the  MNS,  ORD,  RRR,  ILSP 
and  the  D&OTSP. 

10-2.  ILS  Defined 

ILS  is  defined  as  a  disciplined,  unified,  and  iterative  approach 
to  the  management  and  technical  activities  necessary  to: 

a.  Influence  operational  and  materiel  requirements  and  design 
specifications. 

b.  Define  the  support  requirements  best  related  to  a  system 
design  and  to  each  other. 

c.  Develop  and  acquire  the  required  support. 

d.  Provide  the  required  operational  phase  support  at  lowest 
cost. 

e.  Seek  readiness  and  life  cycle  cost  improvements  in  the 
materiel  system  and  support  systems  during  the  operational 
life-cycle 
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f.  Repeatedly  examine  support  requirements  throughout  the 
life  of  the  system. 

10-3 .  ILS  Elements 

ILS  consists  of  12  specific  components,  or  elements. 

a.  Maintenance  Planning. 

b.  Manpower  and  Personnel. 

c.  Supply  Support. 

d.  Support  Equipment  and  Test,  Measurement,  and  Diagnostic 
Equipment  (TMDE) . 

e.  Technical  Data. 

f.  Training  and  Training  Devices. 

g.  Computer  Resources  Support. 

h.  Facilities. 

i.  Packaging,  Handling,  and  Storage. 

j.  Design  Influence. 

k.  Transportation  and  Transportability. 

l.  Standardization  and  Interoperability. 

10-4.  Philosophy  of  T&E  for  Logistics  Supportability 

a.  T&E  of  the  logistics  supportability  will  be  accomplished 
throughout  a  system's  life  cycle  per  AR  700-127.  A 
level-of-repair  analysis  should  be  accomplished  early  in  the  life 
cycle  to  guide  test  planning  for  supportability  issues. 

b.  Early  test  data  will  be  used  to  determine  the  achievement 
of  support-related  design  characteristics,  including  BIT/BITE 
embedded  training.  Early  test  data  will  also  be  used  to 
determine  the  ability  to  meet  system  readiness  objectives  (when 
mature) ,  the  most  effective  logistics  support  for  the  end-item, 
and  the  acceptability  of  the  logistics  support  analysis,  and  to 
assist  in  developing  the  logistics  support  analysis  record 
(LSAR)  . 


c.  Subsequent  testing  and  field  experience  will  be  used  to 
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improve  the  matured  logistics  support  program;  to  update  the 
LSAR;  to  determine  the  effectiveness,  adequacy,  performance,  and 
RAM  of  system-peculiar  support  equipment,  test  program  sets, 
support  software,  and  TMDE;  and  to  update  the  LSAR  and  system 
repair  parts  provisioning  documentation. 

d.  Logistics  supportability  tests  include  logistic 
demonstration  and  formal  evaluation  of  the  system  support  package 
during  testing. 

e.  Supportability  T&E  will  be  conducted  per  AR  700-127  and  DA 
Pam  700-50.  Logistics  supportability  will  be  a  primary  factor  in 
all  program  and  budget  decisions  associated  with  T&E. 

f.  Testers  and  evaluators  will  ensure  that  a  full  range  of 
supportability  characteristics  and  issues  are  developed  and  that 
test  designs  specifically  address  these  characteristics  and 
issues. 


g.  In  all  materiel  acquisition  programs,  the  ILS  effort 
begins  in  the  concept  exploration  and  definition  phase  prior  to 
program  initiation,  continues  throughout  the  entire  acquisition 
cycle,  and  extends  past  the  deployment  phase.  Logistics  testing 
must  therefore  extend  over  the  entire  acquisition  cycle  of  the 
system  and  be  carefully  planned  and  executed  to  ensure  the 
readiness  and  supportability  of  the  system. 


Section  II 

ILS  T&E  Documentation 


10-5.  Documentation  Objectives 

The  main  objective  of  ILS  test  and  evaluation  is  to  verify  that 
the  logistic  support  being  developed  for  the  materiel  system  is 
capable  of  meeting  the  required  objectives  for  both  peacetime  and 
wartime  employment.  ILS  T&E  consists  of  both  DT&E  and  OT&E,  and 
also  includes  post-deployment  supportability  assessments. 

10-6.  Integrated  Logistic  Support  Plan  (ILSP) 

a.  The  ILS  Manager  (provided  by  the  PM/MATDEV)  for  an 
acquisition  system  is  responsible  for  developing  the  ILSP,  which 
is  the  primary  document  for  planning  and  implementing  the  support 
of  the  fielded  system.  It  is  initially  prepared  during  the 
Concept  Exploration  phase,  then  is  progressively  developed  in 
more  detail  as  the  system  moves  through  the  acquisition  phases. 


10-3 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92  (DRAFT) 

) 

b.  Included  in  the  ILSP  is  identification  of  the  specific  ILS 
test  issues  related  to  the  individual  ILS  elements  and  the 
overall 

system  support  and  readiness  objectives. 

c.  The  ILS  Manager  is  assisted  throughout  the  different 
phases  of  the  development  process  by  the  ILS  Management  Team 
(ILSMT) ,  which  is  formed  early  in  the  acquisition  cycle.  The 
ILSMT  is  a  coordination/advisory  group  comprised  of  many  of  the 
TIWG  members,  to,  include  the  IDE,  lOE,  the  testers  (DT  &  OT) , 
logistician,  and  trainer  personnel  from  the  PM/MATDEV  and  CBTDEV 
offices. 

10-7.  Supportability  Assessment  Plan 

a.  Based  upon  the  objectives  as  outlined  in  the  ILSP,  the  ILS 
Manager,  in  conjunction  with  the  ILSMT  (and  in  coordination  with 
the  TIWG) ,  develops  the  Supportability  Assessment  Plan.  The  plan 
identifies  the  testing  approach  and  evaluation  criteria  that  will 
be  used  to  assess  the  supportability-related  design  requirements 
(e.g.,  reliability  and  maintainability)  and  adequacy  of  the 
planned  logistic  support  resources  for  the  system.  Development 
of  the  plan  begins  upon  approval  of  the  ILSP,  and  is  then  updated 
and  refined  as  the  program  progresses  through  the  acquisition 

cycle.  j 

b.  The  ILS  manager  applies  the  techniques  of  Logistic  Support 
Analysis  (LSA) ,  as  described  in  Mil-STD-1388-lA,  to  formulate  the 
ILS  test  and  evaluation  strategy,  establish  the  T&E  program 
objectives  and  criteria  and  identify  required  test  resources.  He 
must  ensure  that  the  ILS  T&E  strategy  is  based  upon  quantified 
supportability  requirements  and  that  the  necessary  quantities  and 
types  of  data  will  be  collected  to  validate  the  various  T&E 
objectives,  both  during  system  development  and  after  deployment 
of  the  system. 

c.  The  T&E  objectives  and  criteria  must  provide  a  basis  upon 
which  to  ensure  that  critical  supportability  issues  and 
requirements  are  resolved  or  achieved  within  acceptable 
confidence  levels. 

10-8.  ILS  and  the  TEMP 

All  planned/required  ILS  T&E  should  be  included  in  the  TEMP.  It 
is  of  critical  importance  that  all  test  resources  required  for 
ILS  testing  be  identified  in  the  TEMP,  in  order  to  ensure  that 
appropriate  resources  are  budgeted  and  allocated  for  testing. 
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Section  III 
Planning  for  ILS  T&E 


10-9.  Planning  Guidelines 

Proper  planning  is  essential  to  the  conduct  of  an  effective  T&E 
program.  The  following  guidelines  will  assist  in  planning  for 
ILS  T&E: 

a.  Develop  a  logistics  test  and  evaluation  strategy  for  the 
overall  program.  This  should  be  performed  in  the  Concept 
Exploration  Phase.  Coordination  during  staffing  of  the  TEMP 
should  include  the  logistics  evaluator  to  determine  when  a 
Logistics  Demonstration  (LD)  (if  needed)  will  be  performed. 

b.  It  is  important  to  understand  that  the  TEMP  is  a  master 
test  planning  document  and  not  a  logistics  test  plan.  The  TEMP 
is  limited  to  30  pages.  A  determination  should  be  made  as  to 
what  tests  can  provide  useful  ILS  data. 

c.  Integrate  ILS  testing  requirements  (where  feasible)  into 
the  existing  DT&E/OT&E  plans.  This  is  accomplished  through  the 
TIWG. 

d.  Identify  all  required  resources,  to  include  test  articles 
and  logistic  support  items  for  DT,  OT,  and  dedicated  ILS  testing. 

e.  In  general,  OT  is  the  primary  source  of  ILS  data.  User 
tests  are  conducted  in  a  realistic  environment  with  personnel 
representative  of  those  who  will  eventually  operate  and  maintain 
the  fielded  system.  Some  LSA  data  can  be  gained  from 
developmental  testing  such  as  early  wear  data,  some 
maintainability  difficulties,  etc. 

f.  Ensure  full  utilization  of  all  data  collected  during  the 
conduct  of  the  T&E  program  to  reduce  dedicated  ILS  testing,  and 
to  assure  maximum  efficiency. 

10-10.  Criteria  for  Evaluation  Planning 

Detailed  evaluation  criteria  for  each  of  the  ILS  elements  listed 
above  are  presented  in  DA  PAM  700-50, "Integrated  Logistic 
Support:  Developmental  Supportability  Test  and  Evaluation  Guide." 


Section  IV 
Conducting  ILS  T&E 
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10-11.  Objectives  for  ILS  T&E 

a.  To  measure  the  supportability  of  a  developing  system 
throughout  the  acquisition  process. 

b.  To  identify  supportability  deficiencies  and  potential 
corrections/ improvements  as  test  data  becomes  available. 

c.  To  assess  the  operational  effectiveness  of  the  planned 
support  system. 

d.  To  evaluate  the  system's  operational  suitability  relative 
to  it's  ability  to  achieve  planned  readiness  objectives. 


10-12.  Makeup  of  ILS  T&E 

ILS  T&E  consists  of  a  series  of  ILS  demonstrations  and 
assessments  which  are  usually  conducted  as  part  of  system 
performance  tests.  Special  end-item  equipment  tests  are  rarely 
conducted  solely  for  ILS  evaluation. 

10-13.  ILS  T&E  Tasks 

Specific  ILS  T&E  tasks  (as  prescribed  in  MIL-STD-1388-1A) 
include: 

a.  Analysis  of  test  results  to  verify  achievement  of 
specified  supportability  requirements. 

b.  Determination  of  improvements  in  supportability  and 
supportability-related  design  parameters  needed  for  the  system  to 
meet  established  goals  and  thresholds. 

c.  Identification  of  areas  where  established  goals  and 
thresholds  have  not  been  demonstrated  within  acceptable 
confidence  levels. 

d.  Development  of  corrections  for  identified  supportability 
problems  such  as  modifications  to  hardware,  software,  support 
plans,  logistic  support  resources,  or  operational  tactics. 

e.  Projection  of  changes  in  costs,  readiness  and  logistic 
support  resources  due  to  implementation  of  corrections. 

f.  Analysis  of  supportability  data  from  the  deployed  system 
to  verify  achievement  of  the  established  goals  and  thresholds 
and,  where  operational  results  deviate  from  projections, 
determination  of  the  causes  and  corrective  actions. 
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10-14.  System  Support  Package  (SSP) 

a.  T&E  of  the  support  for  a  materiel  system  requires  a  SSP 
consisting  of  spares,  support  equipment,  technical  documents  and 
publications,  representative  personnel,  any  peculiar  support 
requirements,  and  the  test  article  itself;  in  short,  all  of  the 
items  that  would  eventually  be  required  when  the  system  is 
operational. 

b.  This  complete  support  package  must  be  at  the  test  site 
before  the  test  is  scheduled  to  begin.  Delays  in  the 
availability  of  certain  support  items  could  prevent  the  test  from 
proceeding  on  schedule  (which  can  be  costly  due  to  on-site 
support  personnel  on  hold  or  tightly  scheduled  system  ranges  and 
expensive  test  resources  not  being  properly  utilized)  or  could 
result  in  the  test  proceeding  without  conducting  the  complete 
evaluation  of  the  support  system. 

c.  The  ILS  test  planner  must  ensure  that  the  required 
personnel  are  trained  and  available,  that  test  facility 
scheduling  has  been  accomplished,  and  that  the  test  support 
package  is  on  site  on  time.  For  more  information  on  the  SSP,  see 
Chapter  9. 

10-15.  ILS  Test  Data 

a.  In  order  to  facilitate  data  sharing  and  continuous 
evaluation,  the  ILS  Manager  should  coordinate  with  the 
appropriate  TIWG  members  to  ensure  that  the  methods  used  for 
collection,  storage,  and  extraction  of  ILS  T&E  data  are 
compatible  with  those  used  in  testing  the  materiel  system. 

b.  As  with  any  testing,  the  ILS  test  planning  must  ensure 
that  all  required  data  are  identified;  that  they  are  sufficient 
to  evaluate  system  readiness  and  supportability ;  and  that  plans 
are  made  for  appropriate  data  classification,  storage,  retrieval, 
and  reduction  necessary  for  analysis. 

10-16.  ILS  Test  Emphasis 

The  emphasis  of  the  ILS  evaluation  changes  as  the  program  moves 
through  the  acquisition  phases.  During  early  phases  of  a 
program,  the  evaluation  results  are  used  primarily  to  verify 
analysis  and  develop  future  projections.  As  the  program  moves 
into  Engineering  and  Manufacturing  Development  and  hardware 
becomes  available,  the  evaluation  addresses  design,  particularly 
the  reliability  and  maintainability  aspects,  training  programs, 
support  equipment  adequacy,  personnel  skills  and  availability, 
and  technical  publications. 
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10-17.  ILS  Shortcomings 

The  ILS  Manager  must  assure  that  logistical  shortcomings  are 
identified  during  the  T&E  process,  and  that  appropriate  action  is 
taken  made  to  correct  deficiencies.  Coordination  with  the  TIWG 
members  is  essential  to  the  conduct  of  meaningful,  efficient  ILS 
T&E. 


Section  V 

ILS  T&E  Approach 


10-18.  ILS  Quantification 

ILS  T&E  must  quantify  the  various  measures  of  ILS  planning, 
logistics  supportability ,  and  operational  suitability.  The  scope 
of  this  effort  is  described  in  the  following  paragraphs. 

a.  The  designated  logistics  support  concepts/system  for  the 
hardware  or  software  systems. 

b.  The  growth  and  maturity  of  relevant  logistics  support 
elements. 

c.  The  impact  of  relevant  logistics  support  elements  on 
system  and  unit  readiness. 

d.  The  logistics  burden  imposed  on  the  unit  and  the  support 
system. 


e.  The  logistics  cost. 

10-19.  Logistics  Supportability  in  the  T&E  Process 
The  logistics  supportability  doctrine,  concepts,  and  organization 
addressed  consider  all  levels  of  organic  maintenance  and  supply 
support.  Figure  10-2  illustrates  the  logistics  supportability 
throughout  various  phases  of  the  T&E  process.  Software 
supportability  is  considered  an  ILS  assessment  item  and  has  to  be 
a  testable  part  of  logistics  testing.  Additional  details  on 
software  are  addressed  in  Part  Seven. 

(INSERT  FIGURE  10-2) 

10-20.  ILS  Assessment  Considerations  and  Documentation 
The  methodology  used  for  logistics  CE  can  be  tailored  to  the  type 
and  level  of  the  system  (ACAT  I  through  IV  program) ,  the  phase  of 
acquisition,  the  acquisition  strategy  (AS) ,  and  the  impact  of  the 
logistics  supportability  on  the  suitability  of  the  system. 
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Figures  10-3  and  10-4  list  the  various  ILS  assessment 
consideration  areas  and  the  documentation  used  or  reguired  in  the 
ILS  process. 


(INSERT  FIGURE  10-3) 

(INSERT  FIGURE  10-4) 

10-21.  ILS  Demonstration  and  Testing 

A  logistics  demonstration  is  required  to  be  conducted  on  most 
(some  ACAT  III  &  IV  programs  do  not)  acquisition  programs  by  the 
PEO/PM/MATDEV.  Contractor  testing,  FDTE,  and  other  testing  are 
also  conducted  as  indicated  in  figure  10-5.  Figures  10-6  through 
10-8  describe  the  evaluator's  role  through  each  of  the 
phases/milestones . 


(INSERT  FIGURE  10-5) 

(INSERT  FIGURE  10-6) 

(INSERT  FIGURE  10-7) 

(INSERT  FIGURE  10-8) 

10-22.  Responsible  Organizations 

The  following  organizations  comprise  the  major  players  in 
logistics  planning  and  analysis  (figure  10-9) .  For  more 
information  about  the  functions  of  these  organizations,  see 
Chapter  3. 


(INSERT  FIGURE  10-9) 

10-23.  Limitations  and  Constraints  on  ILS  OT 

a.  A  system's  ILS  elements  are  combinations  of  generic 
management  processes,  resource  drivers,  and  requirements  that 
should  be  completely  tested,  quantified,  and  evaluated  during  any 
type  acquisition  process.  All  ILS  elements  will  be  considered 
even  though  constraints  may  be  placed  on  the  acquisition  system, 
which  will  require  follow-on  testing  after  initial  fielding  to 
assure  adequate  supportability  exists.  However,  in  no  case 
should  significant,  unresolved  supportability  deficiencies  be 
allowed  to  be  present  at  time  of  fielding. 

b.  ILS  elements  being  assessed  may  require  the  use  of  the  DT 
data  or  contractor  test  data  because  practical  test  constraints 
do  not  realistically  allow  the  impact  of  logistics  supportability 
to  be  portrayed  on  the  receiving  units.  In  these  cases. 
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analysis,  modeling,  or  simulation  techniques  might  be  considered 
to  compensate  for  the  inability  to  perform  the  actual  logistics 
operation. 

c.  Any  constraints  and  limitations  must  be  identified  early 
in  the  acquisition  program.  Early  evaluation  results  in 
identification  of  critical  logistics  supportability  issues, 
criteria,  and  data  sources. 

d.  Simulating  combat  damage  to  determine  logistics  impact  may 
not  be  a  capability  during  OT.  The  recognition  that  combat  damage 
seriously  impacts  war-time  logistics  support  (e.g.,  higher  demand 
of  repair  parts,  ammunition,  and  fuel;  increased  personnel,  and 
special  maintenance  procedures  for  battle  damage  repair  kits  now 
being  established  for  new  system  acquisitions)  must  be  considered 
and  evaluated  carefully  to  assure  that  the  configuration  of ^ 
support  units  can  support  wartime  operational  tempos.  Existing  or 
development  of  new  simulation  models  should  be  considered  in  the 
evaluation  process  to  determine  impact  on  logistics  that  combat 
damage  causes. 

10-24.  Role  of  the  Independent  Evaluator/Assessor  in  ILS 

a.  Independent  development  and  operational  evaluators  are  not 
responsible  for  developing,  applying,  or  managing  the  ILS  process 
and  are  not  the  Army's  overall  evaluator  for  ILS  but  do  have 
specific  responsibility  for: 

(1)  Monitoring  the  LSA  process. 

(2)  Monitoring  the  system  supportability  characteristics 
of  the  system  under  development. 

(3)  Conducting  necessary  DT&E  and  OT&E  to  determine  system 
sustainability  using  the  ILS  concepts  and  doctrine  established  in 
the  acquisition  strategy  for  the  system  being  developed. 

b.  Throughout  the  acquisition  process,  the  evaluators  have  to 
interact  with  members  of  the  ILSMT  and  with  the  independent 
logistician  (DA  DCSLOG)  and  the  MATDEV. 

c.  Perform  on  system  assessment  functions  as  noted  below. 

(1)  Monitor  and  report  the  system's  ILS  element 
deficiencies. 

(2)  Determine  the  support  system's  ability  to  meet  the 
logistics  support  requirements/criteria  established  in  the 
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acquisition  strategy. 

(3)  Establishing  automation  systems  that  describe  in 
detail  all  knowledge  required  to  properly  analyze  impact  of  ILS 
in  T&E. 


Section  VI 
Readiness 


10-25.  Operational  Readiness  Issues 

a.  For  most  systems,  operational  readiness  issues  are 
included  in  T&E  planning  documents.  Operational  readiness  issues 
address  the  capability  and  capacity  of  the  unit  to  achieve  and 
maintain  the  required  peacetime  and  wartime  system  readiness 
objectives  (SROs)  when  the  planned  logistics  support  concepts, 
doctrine,  and  organization  (as  described  in  the  MNS  and  the 
D&OTSP)  and  materiel  (as  described  in  the  SSP)  are  used. 

b.  The  issue  is  normally  limited  to  the  retail  (intermediate 
and  below)  Army  logistics  system.  In  cases  where  two  levels  of 
logistics  support  are  dictated,  such  as  user  and  depot,  the 
operational  readiness  issue  will  include  the  depot  activity. 
Criteria  normally  come  from  the  SRO  in  the  ORD. 

10-26.  System  Readiness  Objectives  (SRO) 

a.  AR  702-3  requires  development  of  an  SRO  for  each  system  or 
item.  For  most  Army  systems,  Ao  has  been  selected  as  the  element 
that  forms  the  SRO. 

b.  For  systems  or  items  like  clothing,  artillery  rounds,  gas 
masks,  or  distributed  systems,  other  measures  such  as  mission 
reliability  or  success,  percent  of  critical  functions  completed, 
or  percent  of  critical  communications  sent  or  received  may  be 
better  indicators  or  measures  of  readiness. 

c.  SRO  are  stated  as  requirements  for  the  IOC  time-frame  and 
are  to  be  supported  by  logistics  supportability  concepts, 
doctrine,  organization,  and  materiel.  The  evaluators  assist  in 
justifying  and  developing  SROs  by  participating  in  the  RRRWG, 

PDRs  and  CDRs. 

d.  The  lEP  or  TEP  shows  how  the  SRO  will  be  estimated,  how 
unit  readiness  will  be  assessed,  and  how  significant  drivers  for 
the  SRO  will  be  determined. 
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e.  A  representative  measure  of  the  SRO  is  seldom  directly 
measurable  from  test.  Test  data  do  not  adequately  reflect 
expected  peacetime  or  wartime  field  conditions.  Certain 
components  of  an  SRO,  such  as  repair  times  and  failure  data,  do 
not  reflect  typical  field  usage.  These  data  require  “adjustment 
through  analysis  before  they  can  be  used  appropriately  in  the 
evaluation.  In  these  situations,  the  evaluator  should  identify 
both  the  demonstrated  and  the  predicted  SRO. 


f.  When  specific  failure  modes  are  corrected  and  verified 
with  little  expectation  that  they  will  reoccur  with  the 
production  item  in  the  field  at  the  same  rate,  the  failure  data 
can  be  revised  in  the  predicted  SRO  estimate. 

g.  When  BITE  has  not  been  developed  and  alternative 
diagnostic  procedures  have  to  be  used  during  a  test  and  are  not 
forecast  to  be  the  eventual  diagnostic  procedures,  an  alternative 
strategy  has  to  be  develop  to  insure  repair  times  should  also  be 
used  to  reflect  the  impact. 

h.  Other  components  of  the  SRO,  such  as  ALDT,  require  more 
substantial  analysis  and  modification,  often  using  assumptions 
and  methodologies  quite  different  from  those  used  in  development 
of  the  criterion. 

10-27.  Unit  Readiness  Analysis 

a.  The  system  and  its  associated  items  of  support  impact  on 
the  unit  readiness  requires  extensive  analysis.  Unit  readiness 
estimates  from  te~t  seldom  reflect  the  realistic  workload  on  the 
unit  because  of  sample  sizes,  equipment  configuration,  and  the 
amount  of  contractor  logistics  support  that  has  to  be  provided 
because  of  the  non-availability  of  organic  support. 

b.  When  testing  requires  dedicated  DS/GS/ intermediate  level 
maintenance  support,  maintenance  workloads  in  test  normally  will 
be  extremely  low  and  will  impact  on  availability  of  legitimate 
data  on  personnel,  facilities,  TMDE,  and  other  elements  of 
support  at  this  logistics  level. 

c.  A  unit  readiness  analysis  based  on  logistics  parameters 
established  during  the  development  process  may  be  extremely 
difficult  and  may  have  to  include  use  of  the  following  data: 

(1)  Engineering  data  that  includes  a  complete  failure 
modes  analysis. 

(2)  All  accumulated  test  data  that  will  include  any  type 
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testing  accomplished. 

(3)  Personnel  and  support  information  that  relates  to  the 
unit  being  evaluated. 

(4)  Current  unit  workload. 

c.  The  unit  readiness  analysis  typically  may  require  a 
computer  model  or  simulation  that  allows  supply,  maintenance, 
transportation,  and  other  ILS  resources  to  be  consumed  and 
replenished  as  the  unit  operates  in  accordance  with  the  planned 
wartime  or  peacetime  OMS/MP.  Modeling  and  simulation  do  not 
replace  the  requirement  to  perform  a  test,  but  only  assist  in 
making  analyses  when  actual  operational  data  do  not  exist. 

d.  Several  existing  models  are  useful  in  this  regard. 

(1)  Army  Unit  Readiness/Sustainability  Assessor  model. 

(2)  Return  to  Combat  model. 

(3)  TRANSANA  Aircraft  Reliability  and  Maintainability 
Simulation. 


e.  These  models  provide  readiness  estimates  such  as  Ao,  days 
of  sustained  operation,  unit  availability,  and  supply  and  support 
resource  consumption.  They  use  engineering  estimates  as  well  as 
test  data  to  address  the  following:  mission  profiles,  support 
resources,  requirements,  personnel  (i.e.,  number  of  personnel  by 
MOS) ,  quantities  of  tools  and  tool  kits,  TMDE,  support  equipment, 
and  supply  support. 

f.  If  a  unit-level  logistics  model  is  planned  for  in  the  lEP 
or  TEP,  the  following  are  done  early  in  the  system's  life  cycle: 


(1)  Identify  the  model  or  characteristics  of  the  model  to 
be 

used. 

(2)  Identify  the  organization  that  will  run  and  maintain 
the  model  and  database. 


(3)  Determine  the  format  and  sources  of  the  required  data. 

(4)  Ensure  that  the  data  are  provided  or  collected  in  the 
correct  format. 


(5)  Ensure  that  the  model  is  accredited. 
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g.  Some  low-density  or  non-repairable  systems  or  items  may 
not  require  a  unit-level  logistic  model.  Other  systems  or  items 
may  require  special-purpose  models  (i.e.,  transportation  or 
supply) . 

10-28.  Administrative  and  Logistics  Downtime  (ALDT) 

a.  Testing  is  normally  not  to  be  delayed  in  order  to  collect 
data  on  realtime  ALDT.  Procedures  for  onsite  repair  of  damaged 
or  contractor ' obtained  parts  are  developed  by  the  test  director 
and  incorporated  into  the  test  plan.  These  procedures  include 
all  anticipated  means  of  acquiring  supply  support  (e.g.,  local 
purchase)  during  the  test. 

b.  During  the  conduct  of  the  test  any  artificial  ALDT  that 
result  from  accomplishing  repairs  or  obtaining  support  that  was 
not  anticipated  in  the  established  SSP  or  required  to  continue 
testing  using  any  unique  support  techniques  will  be  documented  in 
such  a  manner  that  realistic  ALDT  are  established  at  test 
completion  for  evaluator  use. 

10-29.  Adjustment  of  ALDT 

There  are  three  procedures  for  estimating  ALDT  attributable  to 
the  PLL/ASL  (see  figure  10-10) .  The  results  can  be  used  in  the 
Ao  analysis  in  place  of  the  actual  test  data.  The  three 
procedures  are  as  follows: 

(INSERT  FIGURE  10-10) 

a.  ALDT  for  supply  (PLL/ASL)  using  estimates  and  assumptions 
from  the  RAM  Rationale  Report  (AR  702-3)  may  be  calculated  in 
lieu  of  test  data  provided  adequate  justification  is  established. 

b.  When  the  PLL/ASL  for  test  is  complete  and  items  similar  to 
those  being  tested  are  found  on  existing  systems,  order  and  ship 
times  can  be  estimated  from  historical  data.  Given  the  national 
stock  number  (NSN)  and  appropriate  weapon  system  codes,  the  LIF 
of  the  Logistic  Control  Activity  (LCA)  can  provide  order  and  ship 
times  (OST)  for  Europe  and  CONUS.  The  OST  yields  a  more 
representative  ALDT  than  the  test  ALDT  and  is  more  useful  for 
sensitivity  analyses  on  the  SRO. 

c.  When  the  PLL/ASL  for  test  has  incomplete  stockage  levels 
and  the  OST  for  restocking  the  PLL/ASL  are  skewed  because  of  test 
priorities,  the  following  analysis  is  effective  for  making  ALDT 
estimates  more  realistic.  For  each  mission-critical  or  high-cost 
spare  and  repair  part  to  be  analyzed,  construct  a  supply  support 
matrix  similar  to  figure  10-14  using  the  following  methods: 
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(1)  Construct  a  time-line  similar  to  figure  10-10  and 
indicate  the  initial  wartime  and  peacetime  stockage  level  and 
reorder  point  for  each  item. 

(2)  Plot  the  stockage  level  during  the  reorder  period, 
indicating  points  where  demand  occurred  with  zero  stock  balance. 
The  result  will  show  a  more  realistic  ALDT  estimate  that  can  be 
used  to  adjust  for  contractor  repair  on  site,  test  continuation, 
off-system  repair,  and  less  than  100  percent  initial  stockage  of 
the  PLL/ASL. 

(3)  The  following  data  are  required  and  are  available  from 
the  MATDEV: 

(a)  A  listing  of  supply  support  items  on  the  PLL/ASL 

from  the 
SSP. 

(b)  Initial  wartime  and  peacetime  stockage  level  for 
each  item. 

(c)  Mission  criticality  of  each  item. 

(d)  Repair  policy  for  each  item. 

(e)  Resupply  and  replenishment  policy  for  each  item. 

(f)  Reorder  point  for  each  item 

(g)  Location  and  mobility  requirements 

(h)  Item  weight,  volume,  and  size,  and  storage  vans  or 
vehicle  capacity. 

(i)  Sufficient  on-hand  quantities  of  PLL/ASL  items  for 
test  continuity. 

(j)  Supply  support  demands  from  test. 


Section  VII 
Logistics  Burden 


10-30.  Description 

For  each  testable  element  of  the  SSP,  a  logistics  burden  analysis 
is  planned  in  the  lEP  or  TEP  and  executed  in  the  lER  or  TER.  The 
logistics  burden  analysis  compares  support  maintenance,  supply. 
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and  transportation  demands  placed  on  the  support  system  against 
the  resources  planned  for  the  support  system. 

10-31.  ILS  Elements  Addressed 

Each  testable  element  of  the  SSP  is  to  be  addressed  to  determine 
the  strengths  and  weaknesses  of  the  planned  support  in  terms 
meaningful  to  the  decision  process.  As  a  minimum,  the  following 
elements  are  to  be  addressed: 

a.  Maintenance  concept  to  determine  its  relationship  and 
integration  with  Army  logistics  doctrine. 

b.  Supply  support. 

c.  Personnel  and  manpower. 

d.  TMDE. 

e.  Tools,  tool  kits,  and  other  maintenance  equipment. 

f.  Technical  documentation. 

g.  Other  support  equipment. 

h.  Training. 

i.  Facilities  reviews  to  be  assured  support  system  will 
operate  within  planned  construction. 

j.  Packaging,  handling,  and  storage  provisions  with  special 
emphasis  on  ancillary  equipment  that  might  be  required  for 
movement  of  packages. 

10-32.  SSP  Requirements 

The  MATDEV  is  responsible  for  ensuring  that  the  required 
quantities  of  each  element  of  the  SSP  are  delivered  to  the  test 
site.  All  elements  of  the  SSP  are  to  be  provided  in  proportion  to 
the  systems  being  tested.  For  example,  if  6  sets  of 
special-purpose  tools  are  required  for  a  test  of  15  systems,  then 
2  sets  of  special  tools  are  required  for  a  test  of  5  systems. 

10-33.  Logistics  Burden  MOP  , 

MOPs  and  criteria  for  each  element  of  the  SSP  are  required  for 
logistics  burden  analysis  and  evaluation.  When  these  criteria 
have  not  been  developed  by  the  CBTDEV,  the  evaluator  may  be 
required  to  develop  a  set  of  baseline  criteria. ^MOPs  and 
suggested  criteria  are  presented  in  the  discussion  of  each 
support  element. 


10-16 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


Section  VIII 
Supply  Support 


10-34.  Elements  of  Supply  Support  (AR  710-2) 

a.  Selected  spares  and  repair  parts  of  the  PLL/ASL. 

b.  Hands-on  supplies  of  on-board  spares  and  repair  parts. 

c.  BII,  AAL  items,  and  expendable  supplies  and  materiel 
(ESM) . 

d.  POL. 

e.  Ammunition. 

f.  Items  or  media  that  contain  software  (e.g.,  disks,  tapes, 
and  memory  devices) . 

10-35.  Supply  Support  Categories 

To  simplify  tracking  and  analysis  of  the  PLL/ASL  (see  figure 
10-10) ,  supply  support  is  divided  into  the  following  categories: 

a.  Mission-critical  support  (i.e.,  supply  support  necessary 
to  sustain  the  system  in  combat) . 

b.  Non-mission-critical  support. 

c.  Items  common  to  the  unit's  existing  supply  support. 

d.  System-peculiar  items  introduced  into  the  unit's  existing 
supply  support. 

10-36.  Supply  Support  Criteria  and  MOP 

MOPs  and  baseline  criteria  for  supply  support  are  shown  in  table 
10-1.  Data  requirements  are  shown  in  figure  10-11. 

(INSERT  TABLE  10-1) 

(INSERT  FIGURE  10-11) 

10-37.  Supply  Support  Considerations 

In  evaluating  supply  support,  consider  the  following: 

a.  Demand  and  consumption  rates.  Consumption  rates  for 
repair  parts,  spares,  POL,  and  bulk  supplies  are  critical 
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logistics  supportabi lity  readiness  and  cost  indicators.  The 
ability  of  the  supply  system  to  meet  consumption  requirements 
significantly  affects  the  readiness  of  the  fielded  system. 

Compare  stated  criteria  to  the  demand  accommodation  and  ^ 
satisfaction,  direct  exchange,  and  zero  balance  indexes  adjusted 
from  test,  and  overall  supply  availability  rates. 

b.  Replenishment  and  resupply  rates.  Accurate  replenishment 
and  resupply  rate  estimates  for  items  with  demand-oriented 
replenishment  policies  normally  come  from  not  test  data  but  from 
the  MATDEV.  Because  these  rates  may  be  significant  contributors 
to  ALDT,  they  can  be  used  in  system  or  unit  readiness  simulations 
or  models. 

c.  Mobility.  The  addition  of  items  to  the  PLL/ASL  may 
critically  restrict  the  mobility  of  the  PLL/ASL  by  increasing 
volume  without  an  adequate  increase  in  trailers,  trucks,  or  vans. 
In  addition,  items  placed  into  the  PLL/ASL  may  be  sensitive  to 
movement  and  storage  conditions.  The  loading  and  storage  plan 
may  not  be  compatible  with  the  mobility  requirements  or  the  size 
of  new  components.  The  mobility  index  is  measured  in  test  and 
compared  to  the  stated  mobility  index  criteria  or  the  baseline 
criteria. 

d.  Size.  The  addition  of  system-peculiar  supply  support 
items  is  not  to  substantially  increase  the  size  of  the  PLL/ASL. 
Large  increases  in  the  number  or  size  of  line  items  is 
undesirable.  Current  or  planned  storage  capability  may  not 
accommodate  large  components  or  assemblies,  or  may  not  be 
designed  to  adequately  protect  components  that  contain  hazardous 
materials  or  require  special  handling  and  storage.  Storage  space 
for  personal  gear  should  be  available.  The  volume  of  the  PLL/ASL 
is  measured  in  test  and  compared  to  the  available  storage  volume 
using  the  volume  accommodation  index. 

e.  Capacity.  Current  storage  layout/supply  identification 
means  in  the  PLL/ASL  may  not  be  consistent  with  the  new  system. 
Onloading,  offloading,  and  lifting  capacity  may  not  be  sufficient 
for  the  new  items  being  introduced.  Special  handling  and  storage 
requirements  such  as  refrigeration  of  batteries  are  to  be 
addressed.  On-board  spares  and  BII  are  to  be  readily  accessible 
and  not  interfere  with  or  pose  safety  or  health  hazards  to  the 
crew.  The  PLL/ASL  line  item  index  is  to  be  compared  with  stated 
cj^3_'teria.  The  evaluation  addresses  the  need  and  practicality  of 
consumption,  replenishment,  and  storage  of  POL,  on-board  spares, 
ammunition,  repair  parts,  tools,  AAI,  ESM,  and  BII. 


f.  Analysis  constraints. 
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(1)  Preliminary  estimates  of  supply  support  requirements 
are  engineering  estimates  from  the  LSA  process  based  on  projected 
system  usage  rates,  reliability  predictions,  and  component 
failure  rates. 

(2)  The  utility  of  supply  support  data  is  significantly 
limited  by  test  constraints  and  limitations  such  as  duration  of 
the  test,  limited  sample  sizes,  contractor  maintenance, 
engineering  changes  and  immaturity  in  the  Fault  Detection, 
Isolation,  and  Location  System  (FOILS),  and  software. 

(3)  Through  careful  planning  and  use  of  engineering  and 
test  data,  the  PLL/ASL  can  be  simulated  without  interrupting  the 
test.  It  can  be  simulated  manually  or  through  the  use  of  the 
analyses  suggested  in  figures  10-12  and  10-13. 

(INSERT  FIGURE  10-12) 

(INSERT  FIGURE  10-13) 

(4)  It  may  be  advantageous  to  run  a  POL,  ammunition,  or 
PLL/ASL  model  such  as  the  Selected  Essential  Item  Stockage  for 
Availability  Model  or  the  Optimal  Supply  and  Maintenance  Model. 

(5)  Fuel  consumption  data  from  test  may  not  reflect  the 
capability  of  the  unit  to  deliver,  store,  and  dispense  the 
additional  fuel  for  the  new  vehicles.  Use  the  unit's  current 
capability,  the  BOIP,  TOE,  and  MPs  and  consumption  rates  from 
test  data  to  determine  the  additional  capability  required  (e.g., 
fuel  trucks,  storage  tanks,  or  dispensing  equipment)  to 
accommodate  the  new  system. 

(6)  Test  data,  combined  with  engineering  estimates, 
support  the  evaluation  of  supply  support  through  sensitivity, 
readiness,  or  logistics  burden  analyses.  See  figures  10-12  and 
10-13  for  analysis  examples.  Figure  10-14  is  an  example  of  a 
Supply  Support  Matrix. 

(INSERT  FIGURE  10-14) 


Section  IX 

Personnel  and  Manpower 


10-38.  Personnel  and  Manpower  Elements  (AR  602-2) 

Maintenance,  supply,  and  other  support  personnel  and  their 
required  skills  and  training  are  the  elements  of  personnel  and 
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manpower . 

10-39.  MOP  for  Personnel  and  Manpower 

a.  Maintenance  Ratio  (MR)  is  the  ratio  of  maintenance 
manhours  per  operating  hour  or  life  unit.  On  and  off  system  MRs 
are  estimated  for  each  level  of  maintenance  and  MOS.  MR  criteria 
are  typically  found  in  the  requirements  document. 

b.  Mean  Time  to  Repair  (MTTR)  is  estimated  for  each 
maintenance  level  and  for  each  type  of  maintenance  action,  type 
of  repair  (e.g.,  electrical,  mechanical),  or  MOS.  MTTR  criteria 
are  typically  found  in  the  requirements  document. 

c.  Percent  maintenance  actions  at  each  level  of  maintenance. 
Baseline  criteria  can  be  taken  from  the  ALDT  decision  tree  in  the 
RRR. 


10-40.  Data  Requirements  for  Personnel  and  Manpower 
Data  requirements  for  manpower  and  personnel  are  given  in  figure 
10-11. 

10-41.  Evaluation  Considerations  for  Personnel  and  Manpower 
In  evaluating  manpower  and  personnel,  consider  the  following: 

a.  Estimate  the  expected  workload  for  each  operator, 
maintainer,  supply,  transportation,  and  other  support  personnel 
MOS,  based  on  test  data.  Include  estimates  of — 

(1)  Preventive  maintenance  checks  and  services. 

(2)  Unscheduled  maintenance  to  include  battle  damage 
repair  procedures  using  the  Battle  Damage  Repair  Kits. 

(3)  Operator,  supply,  transportation,  and  other  support 
tasks  not  experienced  in  test  (e.g.,  quarterly  services). 

b.  Compare  these  estimates  against  those  identified  in  the 
QQPRI  (see  figure  10-4)  to  identify  shortfalls.  This  method 
validates  the  adequacy  of  both  the  maintenance  level  at  which  the 
tasks  have  been  assigned  and  the  MOS  selected. 

c.  For  MR,  MTTR,  and  percent  maintenance  actions  at  each 
maintenance  level,  compare  the  stated  criteria  from  the 
requirements  document  to  the  adjusted  estimates  from  test. 

d.  Because  most  test  maintenance,  supply,  and  transportation 
personnel  are  dedicated  to  the  test,  data  collected  on  manpower 
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and  personnel  at  higher  than  organizational  level  are  not  likely 
to  provide  a  realistic  estimate  of  workload. 

e.  T&E  of  manpower  and  personnel  supports  MANPRINT  efforts 
and  vice  versa.  See  figure  10-15  for  an  analysis  example. 

(INSERT  FIGURE  10-15) 


Section  X 
TMDE 


10-42.  TMDE  Elements  (AR  750-43,  MIL-STD-1309) 

a.  All  common  or  general-purpose  manual  test  equipment  and 
ATE.  ATE  can  measure  selected  parameters  of  an  item  using  test 
program  sets  that  satisfy  functional  tests  for  those  parameters. 

b.  Selected  special-purpose  TMDE. 

c.  The  simplified  test  equipment  series  of  ATE  such  as  the 
internal  combustion  engine  or  expandable  (STE-X)  at  the 
organizational  level. 

d.  The  intermediate  forward  test  equipment  composed  of  a 
contact  test  set,  base  station  test  facility,  and  an 
electro-optical  test  facility  at  the  intermediate  level. 

e.  The  AN/MSM-105 (V) 2  and  AN/USM-410 (V) 2  as  found  throughout 
DS/GS  or  Intermediate  and  above  levels  of  maintenance. 

f.  TPSs,  BITE,  and  calibration  equipment. 

10-43.  TMDE  Requirements 

TMDE  may  be  acquired  under  a  separate  requirements  document  or  in 
a  separate  annex  to  the  supported  end  item  requirements  document. 
In  either  case,  it  has  its  own  performance,  RAM,  and  logistics 
requirements.  Formal  procedures  have  been  established  for 
justifying  and  acquiring  special-purpose  TMDE.  Each  TMDE 
requirements  document,  PIP,  and  TMDE  annex  to  the  supported  end 
item  has  an  MNS  concept. 

10-44.  BIT/BITE 

BITE  is  used  in  fault  detection,  isolation,  or  location  and 
involves  digital  or  analog  signals,  warning  and  advisory 
messages,  lights,  audio  signals,  or  switches.  It  is  usually 
planned  to  detect,  isolate,  or  locate  a  percentage  of  system 
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faults  to  a  specific  ambiguity  group  level,  LRU,  or 
shop-replaceable  unit  (SRU)  (see  glossary  for  definitions) .  The 
evaluation  examines  BITE  effectiveness,  software,  and  growth 
during  system  development.  BITE  data  requirements  are  to  be 
included  in  instrumentation  requirements. 

10-45.  Testing  of  TMDE 

a.  TMDE  is'  allocated  to  the  maintenance  levels  as  specified 
in  the  MAC  (see  figure  10-3).  All  manual,  general-purpose  TMDE, 
and  ATE  with  TPSs  will  be  available  as  part  of  the  SSP  to  be  used 
during  test. 

b.  The  test  will  evaluate  all  TMDE  to  determine  that  its 
capabilities  during  the  required  logistics  demonstration  can  be 
met  in  the  operational  environments  scheduled  for  the  equipment 
use.  This  will  require  special  exercises  to  include 
insertion  events  in  order  to  fully  evaluate  TMDE  capabilities. 

c.  All  TMDE,  ATE  with  complete  TPSs  (including  software, 
interconnecting  devices,  and  documentation)  and  BIT/BITE  will  be 
sufficiently  complete  to  allow  full  utilization  during  lOTE. 

10-46.  TMDE  Analysis 

See  figure  10-16  for  an  analysis  example.  MOPs  for  TMDE  are  as 
follows  (no  baseline  criteria  are  provided) : 

(INSERT  FIGURE  10-16) 

a.  Probability  of  correct  fault  detection.  This  probability 
calculation  is  the  ratio  of  fault  indications  (correctly  isolated 
to  some  specified  ambiguity  group,  LRU,  or  SRU)  to  the  number  of 
confirmed,  incorrect,  or  absent  fault  indications.  It  is 
calculated  for  each  type  of  BITE,  TMDE,  or  TPS. 

b.  Probability  of  correct  fault  isolation  and  location.  This 
calculation  is  the  ratio  of  confirmed  faults  (correctly  located 
or  isolated  to  a  specified  ambiguity  group,  LRU,  or  SRU)  to  the 
number  of  location  or  isolation  attempts.  It  is  computed  for 
each  type  of  BITE,  TMDE,  or  TPS  and  includes  the  measurement  of 
false  indications. 

c.  TMDE  utilization  rates.  This  rate  is  a  ratio  of  the 
utilization  hours  of  a  piece  of  TMDE  to  the  operating  hours  of 
the  supported  end  item. 

d.  Percent  faults  detected/ located  by  manual,  semiautomatic, 
and  automatic  techniques. 
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e.  Time  supportability.  This  factor  is  the  ability  of  the 
TMDE  to  be  supported.  It  includes  the  TMDE  RAM  characteristics 
and  adequacy  of  the  system  support  package  required  for  the  TMDE. 

10-47.  Data  Requirements  for  TMDE 

Data  requirements  for  TMDE  are  shown  in  figure  10-11. 

10-48.  Evaluation  Considerations  for  TMDE 

a.  Evaluate  the  operational  effectiveness,  suitability,  and 
supportability  of  TMDE  both  quantitatively  and  qualitatively. 

b.  Compare  the  stated  criteria  to  the  percent  correct  fault 
detection  or  percent  correct  fault  isolation  or  location  from 
test  or  LD. 

c.  Compare  TMDE  RAM  against  the  stated  requirements. 

d.  Identify  TMDE  utilization  rates  for  each  piece  of  TMDE. 

e.  Address  percentage  of  faults  detected,  isolated,  or 
located  by  manual,  semiautomatic,  and  automatic  procedures. 

f.  These  are  some  of  the  questions  that  should  be  answered: 

(1)  Are  sufficient  quantities  of  TMDE  located  at  the 
proper  maintenance  levels? 

(2)  What  is  the  increase  in  common  and  peculiar  manual 
test  equipment  and  ATE  over  the  unit's  current  TMDE? 

(3)  Are  the  TPSs  compatible  with  existing  ATE? 

(4)  will  the  TPSs  be  validated  and  verified  prior  to 
fielding? 

(5)  Are  adequate  storage  areas  being  provided  for  the  TPS? 


Section  XI 

Tools,  Tool  Kits,  and  Other  Maintenance  Equipment 


10-49.  Elements  of  Tools,  Tool  Kits,  and  Other  Maintenance 
Equipment 

Included  are  common  and  special  tools  or  tool  kits,  jigs, 
fixtures,  stands,  lifting  devices,  and  any  other  type  of 
servicing  (lubrication  guns,  pumps)  or  maintenance  equipment. 
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10-50.  MOP  for  Tools,  Tool  Kits,  and  Other  Maintenance  Equipment 
MOPS  for  tools,  tool  kits,  and  other  maintenance  equipment  are 
given  in  Table  10-1.  Similar  measures  can  be  applied  to  any 
maintenance  equipment.  A  design  goal  is  the  elimination  or 
minimum  use  of  special  tools  or  tool  kits,  jigs,  fixtures, 
stands,  and  lifting  devices  and  maximum  use  of  existing 
equipment.  If  new  items  are  required,  they  are  to  minimize 
training  and  storage  requirements.  A  reduction  in  reliance  on 
common  items  is  also  a  design  goal. 

10-51.  Data  Requirements  for  Tools,  Tool  Kits,  and  Other 
Maintenance  Equipment 

Data  requirements  for  tools,  tool  kits,  and  other  maintenance 
equipment  are  given  in  figure  10-12. 

10-52.  Evaluation  Considerations  for  Tools,  Tool  Kits,  and  Other 
Maintenance  Equipment 

a.  Adequacy  of  all  special  and  common  tools,  tool  kits,  and 
other  maintenance  equipment  needed  to  perform  critical  tasks  and 
procedures,  including  special  tools  listed  in  the  RPSTL  BII,  and 
tools  allocated  to  each  maintenance  level  listed  in  the  SSP,  . 

b.  Location  at  the  appropriate  level  of  maintenance. 

c.  Availability. 

d.  Effectiveness. 

e.  Sufficiency  of  quantities  provided. 

f.  Ensure  that  all  critical  operator,  maintenance,  and 
support  tasks  and  procedures  can  be  done  using  the  appropriate 
special  and  common  tools  and  tool  kits. 

g.  Evaluate  work  stands,  fixtures,  jigs  and  any  other  type  of 
servicing  (lubrication  guns,  pumps)  or  maintenance  equipment  with 
respect  to  ease  of  use,  operating  instructions,  standardization, 
and  adequacy  of  numbers  and  location  at  each  maintenance  level. 

h.  Tools  that  support  a  particular  system  are  often 
consolidated  into  tool  kits.  Special  tools,  even  though  part  of  a 
tool  kit,  are  evaluated  separately. 

i.  When  necessary,  tailor  the  evaluation  for  specific 
applications  (i.e.,  special  tools  only  or  a  subset  of  a 
particular  tool  kit) .  See  figure  10-17  for  an  analysis  example. 
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(INSERT  FIGURE  10-17) 


Section  XII 

Technical  Documentation 


10-53.  Elements  of  Technical  Documentation 

Technical  documentation  includes  all  of  the  following  manuals: 
operator,  intermediate,  direct  support,  general  support 
maintenance,  supply  support,  calibration,  handling,  storage,  and 
transportation.  Also  included  are  separate  documents  on  specific 
maintenance,  special  inspections,  lubrication,  or  other 
instructions . 

10-54.  MOP  for  Technical  Documentation 

MOPS  for  technical  documentation  are  given  in  table  10-1. 

10-55.  Data  Requirements  for  Technical  Documentation 
Data  requirements  for  technical  documentation  are  in  figure 
10-12. 

10-56.  Evaluation  Considerations  for  Technical  Documentation 

a.  Evaluating  manuals  consists  of  two  distinct  tasks.  Unless 
the  two  tasks  are  done  separately,  it  may  be  impossible  to 
determine  if  the  manual  is  in  error  or  if  the  user  followed  the 
procedures  incorrectly.  The  two  tasks  are 

(1)  Determine  if  the  drawings,  figures,  specifications, 
and  procedures  are  technically  correct.  Do  this  task  during  TT, 
MDs ,  and  LDs . 

(2)  Determine  if  the  soldier  can  understand  and  correctly 
perform  the  procedures.  This  is  a  most  important  task  for  OT. 
This  task  includes  ensuring  that  tools,  TMDE,  support  equipment, 
supply  support,  and  critical  tasks  are  allocated  by  the  manuals 
to  the  correct  level  of  maintenance  and  MOS. 

b.  As  a  minimum,  all  critical  tasks  and  procedures  in  the 
operators  and  intermediate/DS-GS  maintenance  manuals  and  in  the 
supply,  calibration,  storage,  handling,  or  transportation  manuals 
or  supporting  documentation  are  to  be  verified.  Operation, 
maintenance,  and  support  actions  that  occur  naturally  in  test  may 
be  limited  and  not  represent  the  total  set  of  critical  tasks  and 
procedures  identified  in  the  critical  task  listing.  Any  critical 
task  or  procedure  that  is  not  demonstrated  in  the  logistics  or 
maintenance  demonstration  or  that  does  not  occur  naturally  in  the 
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OT  will  be  verified  in  a  separate  exercise  after  the  OT. 

c.  In  evaluating  technical  documentation,  cover  the  following 
points: 

(1)  Users'  ability  to  locate  procedures  and  tasks  in  the 
manual. 

(2)  Clarity  and  accuracy  of  the  manuals. 

(3)  Need  for  additional  tasks  to  be  included  in  the 
manual. 

(4)  Validity  of  tasks  and  procedures  that  rely  on  ATE  and 
that  are  included  in  backup  manuals. 

(5)  MAC  time  estimates  compared  to  demonstrated  times. 

(6)  Ready  visibility  of  cautions,  warnings,  and  advisories 
in  the  manual. 

(7)  Validation  of  and  necessity  for  preventive  maintenance 
checks,  services,  and  procedures  and  scheduled  services. 

d.  Software  documentation  is  addressed  as  a  separate  item 
because  of  its  criticality. 

e.  See  figure  10-18  for  an  analysis  example. 

(INSERT  FIGURE  10-18) 


Section  XIII 

Other  Support  Equipment 


10-57.  Elements  of  Other  Support  Equipment 

Other  support  equipment  includes  generators;  trucks,  trailers, 
and  transportation  and  handling  equipment;  shop  and  supply  vans; 
retrieval  and  resupply  vehicles;  calibration  vehicles;  ammunition 
and  fuel  trucks;  and  bridges. 

10-58.  MOP  for  Other  Support  Equipment 

No  baseline  MOPs  are  provided.  Demand  is  to  be  accommodated  by 
the  amounts  stated  in  the  BOIP. 

10-59.  Data  Requirements  for  Other  Support  Equipment 

Data  requirements  for  other  support  equipment  are  given  in  figure 
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10-12. 

10-60.  Evaluation  Considerations  for  Other  Support  Equipment 
To  evaluate  support  equipment  (both  old  and  new),  compare  test^ 
data  against  amounts  stated  in  the  BOIP.  Adequate  early  planning 
will  cause  sources  other  than  the  OT  to  generate  much  of  the 
required  data.  In  evaluating,  address  the  following  points. 

a.  The  following  can  be  demonstrated  in  TT  using  military 
test  players: 

(1)  Peculiar  repair  requirements  such  as  grounding  straps 
for  electronic  repair. 

(2)  Environmental  storage  adequacy. 

(3)  Loading,  unloading,  and  storage  capacity  of  the 
PLL/ASL. 

(4)  Onloading  and  offloading  on  all  authorized 
transportation  modes. 

(5)  Tiedown  and  transportation  capability. 

(6)  Compatibility  of  the  fuel  dispensing  system  (i.e., 
nozzles,  pumps,  and  hoses) . 

b.  Analyze  the  adequacy  of  fuel  truck  increases  planned  in 
the  BOIP  by  using  the  OMS/MP  and  fuel  consumption  data  from 
contractor  tests. 

c.  Verify  the  capacity  of  a  tractor  trailer  to  transport  the 
system  by  comparing  the  dimensions,  gross  weight,  and  peculiar 
transportation  requirements  (i.e.,  tiedown  limitations)  of  the 
system  to  the  dimensions  of  the  tractor  trailer  and  its 
load-hauling  capacity. 

d.  Evaluate  ammunition  and  retrieval  vehicles,  maintenance 
support,  and  handling  equipment  to  ensure  that  they  support  their 
intended  functions. 

e.  Determine  the  capacity  of  the  supply  trucks  or  vans  to 
store  and  transport  supply  support  by  analyzing  the  volume  of  the 
PLL/ASL,  the  space  available  in  the  existing  vehicles,  and  the 
increase  in  the  number  of  vehicles  from  the  BOIP.  Supply  vans 
and  trucks  are  to  be  organized  for  easy  identification  of  repair 
parts,  spares,  and  other  items  of  support. 
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f.  The  dimensions  of  a  component  part  may  not  be  compatible 
with  the  proposed  layout  and  may  require  more  space  than  planned. 


g.  Allocated  workspace  for  the  maintenance  personnel  is  not 
to  interfere  with  the  ability  of  personnel  to  accomplish  required 
tasks.  Commercial  graphical  computer  techniques  are  available 
that  can  show  the  configuration  and  layout  of  the  shops  and  vans. 
Storage  space  for  personnel  gear  is  to  be  allocated. 


Section  XIV 
Training 


10-61.  Elements  of  Training 

Training  (AR  350-35)  includes  training  aids,  simulators,  training 
materials,  instructors,  and  on-the-job  training. 

10-62.  MOP  for  Training 

MOPS  for  training  include  "critical  tasks  demonstrated."  This 
MOP  is  the  ratio  of  critical  tasks  demonstrated  by  the  soldier 
using  validated  procedures  within  the  time  standard,  to  the  total 
number  of  tasks  attempted  or  total  tasks  in  the  manuals.  It  can 
be  calculated  for  each  maintenance  level  or  MOS.  Other  MOPs  for 
training  are  developed  under  MANPRINT. 

10-63.  Data  Requirements  for  Training 

Data  requirements  for  training  are  collected  under  manpower  and 
personnel  and  MANPRINT. 

10-64.  Evaluation  Considerations  for  Training 

a.  As  a  minimum,  evaluate  the  effectiveness  of  the  TRSP  with 
respect  to  any  existing  or  new  critical  tasks  required  of  the 
maintenance,  operation,  transportation,  supply,  calibration,  or 
support  personnel  or  the  contact  teams.  As  a  minimum,  the 
operators  and  maintainers  are  required  to  perform  all  the 
critical  operator  and  maintainer  tasks. 

b.  Address  unit  and  individual  training  through  the 
intermediate  and  DS/GS  level. 

c.  Address  ICTPs,  POIs,  and  any  other  training-related 
documentation,  devices,  and  aids. 

d.  When  possible,  plan  for  maximum  use  of  data  from  training 
simulators,  mockups,  and  other  innovative  training  concepts. 
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Section  XV 
Transportability 


10-65.  Elements  of  Transportability 

a.  Transportability  (AR  70-44,  AR  70-47)  includes  the  ability 
to  move  the  system  into  a  theater  of  operations-strategic,  and^ 
move  it  within  the  theater  of  operations-tactical  consistent  with 
the  mission.  This  element  may  deal  with  airplane,  train,  or  ship 
loading  and  internal  or  external  helicopter  loads.  This  focus 
will  allow  the  evaluator  to  determine  if  the  system  is 
deployable. 

b.  Transportability  is  a  major  consideration  in  the  T&E  of 
Army  systems,  including  system  components  and  spare  parts.  T&E 
of  transportability  will  address  the  end-item  in  its  tactical  and 
packaged  or  shipping  configurations,  as  well  as  associated 
support  equipment  and  TMDE. 

c.  Transportability  T&E  is  required  as  part  of  developmental 
testing.  The  ability  of  a  system  to  withstand  the  expected 
transport  environment  over  the  useful  life  of  the  system  must  be 
demonstrated  T&E  before  the  production  decision. 

Transportability  characteristics  of  systems  will  be  assessed  and 
included  in  the  developmental  lER. 

d.  It  is  also  appropriate  to  evaluate  transportability  during 
OT.  Soldiers  that  normally  prepare  the  system  for  movement 
should  be  used  during  these  tests  under  realistic  conditions. 

10-66.  MOP  for  Transportability 

MOPs  for  transportability  may  be  questions  of  the  following 
types : 

a.  Can  the  system  be  transported  to  the  theater  by  the 
preferred  means? 

b.  Are  assets  available? 

c.  Can  the  system  be  moved  adequately  within  the  theater  of 
operations? 

d.  Are  dimensions  and  weight  under  required  limits? 

10-67.  Evaluation  Considerations  in  Transportability 
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a.  Evaluate  not  only  the  ability  to  carry  the  load,  but  also 
the  availability  of  the  mode  of  transportation. 

b.  Also  ensure  that  the  weight  and  dimensions  of  the  new 
system  can  be  supported  by  the  current  bridging  (including 
tactical  bridging)  and  transportation  network  in  the  required 
operational  environment. 

c.  For  large  systems  such  as  vehicles  the  major  source  of 
evaluation  information  for  transportability  is  MTMC.  As  the 
Army's  transportability  agent,  MTMC  provides  transportability 
approvals  or  recommendations  for  correcting  deficiencies  on  new 
systems . 


d.  Most  of  the  airlift,  sealift,  and  rail  transportation 
requirements  are  documented  in  AR  70-47.  The  evaluator  should 
ensure  MTMC  or  other  approved  agency  conducts  a  transportability 
assessment.  For  smaller  systems  the  analysis  may  consist  of 
assessing  the  unit's  capability  to  carry  the  new  system  in 
addition  to  the  required  load. 


Section  XVI 
Logistics  Cost 


10-68.  Requirements  for  Logistics  Cost  Analysis 
Cost  requirements  and  considerations  are  mandated  by  AR  702-3, 
DODI  5000.2,  and  the  VCSA  message  of  April  1986  revising  RAM 
policy.  The  independent  operational  evaluator  incorporates  a 
logistics  supportability  cost  issue  in  the  TEP  when  logistics 
supportability  or  operation  and  support  (O&S)  cost  criteria  exist 
in  the  requirements  document  or  when  high-cost  supportability 
considerations  are  identified. 

10-69.  Logistics  Cost  Criteria 

Criteria  for  the  logistics  supportability  issue  are  stated  as 
logistics  support  or  O&S  cost  parameters  and  are  compatible  with 
the  cost  criteria  in  the  requirements  document.  These 
requirements  are  to  be  stated  (in  dollars)  as  cost  per  repair  or 
repair  part,  or  as  maintenance  manpower  cost  per  year  per 
division. 

10-70.  Logistics  Cost  Methodology 

a.  Logistics  supportability  and  O&S  costs  are  major 
considerations  during  system  design,  development,  and  acquisition 
and  emphasis  is  on  the  reduction  of  these  costs  before  and  after 


10-30 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


system  fielding.  The  lEP  or  TEP  describes  how  the  cost  criteria 
will  be  estimated  from  T&E  data. 

b.  Cost  data  (in  dollars)  usually  come  from  engineering 
estimates  provided  by  the  MATDEV.  In  addition,  the  independent 
operational  evaluator  identifies  and  reports  any  potential 
high-dollar  logistics  support  or  O&S  costs  uncovered  during 
system  T&E.  These  cost  considerations  are  identified  and  reported 
in  the  lER.  For  example — 

(1)  Evaluation  may  reveal  need  for  more  supply  vans  than 
identified  in  the  BOIP.  This  increase  may  indicate  a  substantial 
investment  cost  for  the  Army. 

(2)  Test  data  may  indicate  that  high  dollar  repair  parts 
are  experiencing  failure  rates  higher  than  anticipated  or  that 
substantially  more  maintenance  personnel  of  a  particular  MOS  are 
required. 

Q,  Figure  10—19  is  an  example  of  an  analysis  for  determining 
logistics  costs. 

(INSERT  FIGURE  10-19) 


Section  XVII 

ILS  Considerations  in  Test  Planning 


10-71.  Test  Concept  ^  ^  ,  4.  4. 

The  test  concept  addresses  the  planning  necessary  to  fully  test 
and  evaluate  the  SSP,  how  the  logistics  support  concepts  and 
doctrine  will  be  employed  during  the  test,  the  type^  level,  and 
degree  of  realism  needed  to  adequately  demonstrate  the  support 
concept  is  suitable  to  the  Army  in  combat,  and  the  test 
conditions  or  factors  that  may  affect  the  logistics 
supportability  of  the  system  (e.g.,  ALDT) .  Figure  10—20  is  a 
sample  of  a  data  source  matrix  for  logistics  issues. 

(INSERT  FIGURE  10-20) 

10-72.  Logistics  Support  Guidelines  for  Test 
During  conduct  of  the  test,  the  logistics  support  concepts, 
doctrine,  organization,  and  materiel  are  to  be  played  as 
realistically  as  possible.  Unit  SOPs  for  maintenance  and  support 
developed  and  incorporated  into  the  test.  The  following  test 
elements  are  to  represent  the  anticipated  field  environment  to 
the  maximum  extent  possible: 
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a.  Maintenance  and  support  facilities. 

b.  Location  of  supply,  transportation,  and  support  personnel 
and  equipment. 

c.  Maintenance,  supply,  and  support  tools  and  tool  kits, 
jigs,  fixtures,  stands,  lifting  devices,  and  pumps. 

d.  Use  of  contact  and  calibration  teams.  If  contact  teams  are 
part  of  the  maintenance  concept,  they  are  to  be  played  in  test 
with  the  appropriate  vehicles,  maintenance,  and  test  equipment. 

10-73.  Selective  Logistics  Play 

At  direct  support  and  higher  maintenance,  sometimes  only  selected 
functions  of  supply  and  support  may  be  played  realistically  (see 
figure  10-21  for  an  example  of  typical  logistics  functions) . 
Intermediate,  Direct,  and  General  support  maintenance  personnel 
are  played  with  an  evaluation  made  even  if  they  are  dedicated  to 
the  systems  in  test.  If  all  levels  of  support  are  realistically 
played,  additional  supportability  testing  will  be  done. 

(INSERT  FIGURE  10-21) 

10-74.  System  Support  Package  Testing 

TT  and  OT  will  not  be  initiated  unless  a  complete  SSP  is 
available  or  formal  SSP  waiver  has  been  granted.  Waiver 
procedures  for  the  SSP  are  provided  in  AR  700-127.  The 
regulation  states  that  using  the  SSP  solely  on  an  "as  needed 
basis”  to  support  the  system  under  test  is  not  acceptable.  The 
SSP  is  considered  part  of  the  system  under  test.  The  DTD  for 
logistics  inventories  the  SSP  and  notifies  the  independent 
evaluator  and  MATDEV  of  deficiencies  or  missing  items. 

10-75.  Supply  System  Testing 

Supply  support  is  to  be  representative  of  the  military  support 
system  unless  precluded  by  test  or  test  unit  limitations.  In 
those  cases  where  the  contractor  maintains  the  supply  support 
system,  complete  data  collection  will  be  accomplished  to  allow 
for  evaluation.  The  test  organization  assures  that  the 
contractor's  input  does  not  unfavorably  affect  the  established 
logistics  support  concept,  and,  the  supply  support  concept 
mirrors  the  established  logistics  support  concept  approved  for 
the  system. 

10-76.  Support  Equipment  Testing 

Support  equipment  and  personnel  are  provided  as  part  of  the  SSP 
in  proportion  to  the  systems  under  tests.  Recovery,  refuel, 
rearm,  transportation,  and  other  procedures  and  vehicles  for  test 


10-32 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92  (DRAFT) 


are  to  be  identified  in  the  lEP  or  TEP  and  will  be  generally  be 
part  of  the  SSP  if  the  equipment  being  tested  requires  these 
functions  to  operate  efficiently  under  combat  conditions. 

10-77.  Support  Package  Completeness 

Before  test,  the  MATDEV  (per  AR  700-127)  provides  a  list  of  the 
peculiar  mission-critical  spares  and  repair  parts  of  the  PLL/ASL, 
including  planned  stockage  levels,  replenishment  policies,  and 
quantities  of  items  for  test.  One  hundred  percent  stockage  of 
the  PLL/ASL  is  not  required  for  test,  if  accounting  procedures 
are  established  to  reconstruct  performance  of  the  PLL/ASL  at  100 
percent  stockage.  However,  sufficient  stockage  levels, 
replenishment  policies,  spares,  and  repair  parts  are  required  for 
test  continuity.  Bulk  items  like  common  bolts,  tape,  or 
fasteners  cannot  be  efficiently  addressed  in  test. 

10-78.  Maintenance  Considerations  in  Test 

a.  During  lOTE  or  activities  that  provide  input  for 
consideration  during  full-production  decisions  for  major 
acquisition  programs,  the  system  contractor  personnel  cannot 
participate  except  to  the  extent  that  they  will  be  involved  in 
the  operation,  maintenance,  and  other  support  of  the  system  when 
it  is  deployed. 

b.  If  contractor  maintenance  is  part  of  the  maintenance 
organization  and  concept,  it  can  be  played  in  test.  When  the 
contractor  is  involved  in  system  maintenance,  procedures  are  to 
be  developed  to  exercise  and  control  these  maintenance  functions. 
The  soldier  maintenance  and  support  personnel  will  be  afforded 
the  opportunity  to  maintain  and  support  the  system  without 
contractor  interference.  At  no  time  should  the  contractor  be 
involved  in  the  maintenance  or  support  of  the  system  unless 
authorized  by  the  TEP  or  by  the  SSP. 

c.  Use  of  contractor-peculiar  items  that  will  not  become  part 
of  the  support  package  to  be  fielded  will  not  be  used  unless 
specifically  approved  by  the  TEP  to  ensure  that  the  function  the 
peculiar  equipment  is  required  for  is  included  in  the  LSA/LSAR. 

If  peculiar  contractor  items  are  used,  they  must  be  authorized  in 
the  technical  data  package. 

10-79.  Controlled  Substitution  in  Test 

Controlled  Substitution,  formerly  described  as  cannibalization, 
is  not  an  acceptable  means  of  obtaining  support  during  OT  and  is 
used  only  when  all  other  alternatives  are  exhausted  and 
unacceptable  delay  of  test  results.  If  controlled  substitution  is 
done,  it  is  to  be  done  under  the  control  of  the  DTD  for 
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logistics.  Data  collection  procedures  are  to  be  developed  to 
specifically  identify  and  document  all  substituted  parts  and 
systems  with  adequate  reporting  procedures  established  for  the 
evaluator's  use  after  test. 

10-80.  Logistics  Supportability  Testing  Deficiencies 

a.  Sometimes  because  of  test  schedule  constraints,  shortage 
of  some  SSP  items,  and  unavailability  of  the  prime  test  item, 
some  of  the  sustainment /supportability  functions  may  be  delayed 
or  postponed  during  some  periods  of  OT.  Critical  tasks,  TMDE, 
and  other  support  equipment  may  not  be  demonstrated  with  soldier 
interaction  during  the  test. 

b.  When  this  occurs  during  the  OT  test  period,  the  DTD  for 
logistics  will  require  that  the  CAPSTONE  logistics  functions 
necessary  to  properly  evaluate  the  system's  sustainability/ 
supportability  be  tested  prior  to  the  completion  of  OT.  These 
tests  are  not  to  be  considered  as  additional  tests  or  exercises 
but  required  to  insure  that  all  critical  supportability  tasks 
have  been  investigated.  These  will  normally  involve  the 
demonstration  of  critical  tasks  and  diagnostic  procedures  by 
using  inserted  faults,  removals,  teardowns,  and  use  of  special 
tools  or  TMDE  by  soldier.  The  DTD  for  logistics  determines  when 
such  delayed  or  postponed  tests/exercises  have  been  completed 
based  on  the  emerging  results  of  OT  process. 

c.  If  it  is  determined  by  the  DTD  for  logistics  that 
inadequate  supportability  testing  has  been  performed  or  is  not 
scheduled,  the  DA  DCSLOG'S  independent  logistician  (AMSAA)  will 
be  advised  and  a  risk  analysis  be  accomplished  and  presented  at 
the  milestone  decision  review.  This  will  include  all  acquisition 
systems . 
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Continuous  evaluation  objectives  for  logistics  supportability . 
The  CE  objectives  for  logistics  supportability  are  to — 

a.  Ensure  that  the  ILS  assessment  considerations  are 
compatible  with  any  type  of  operational  testing. 

b.  Identify,  track,  and  report  logistics  supportability 
deficiencies  and  shortcomings. 

c.  Test  the  system's  logistics  support  concepts,  doctrine, 
organization,  and  hardware  and  ancillary  materiel  in  the 
intended  operational  environment. 

d.  Provide  continuous  evaluation  of  the  system  throughout  its 
life  cycle  and  provide  data  found  during  the  CE  process  to 
Army  Activities  involved  with  the  system. 


Figure  10-1.  ILS  Continuous  Evaluation  Objectives. 
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Figure  10-2  (cent).  Logistics  supportability  in  the  T&E  process. 
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Figure  10-2.  Logistics  supportability  in  the  T&E  process. 


ILS  ASSESSMENT  CONSIDERATIONS; 


(1)  Design  Influence. 

(2)  Maintenance  Planning. 

(3)  Manpower  and  Personnel. 

(4)  Supply  Support. 

(5)  Packaging,  handling,  and  storage. 

(6)  Support  Equipment  and  TMDE. 

(7)  Training  and  Training  Devices. 

(8)  Technical  Data. 

(9)  Computer  Resources  Support. 

(10)  Transportation  and  Transportability. 

(11)  Facilities. 

(12)  Standardization  and  Interoperability. 

(13)  RAM. 

(14)  Support  Management  and  Analysis. 

(15)  Cost  Analysis  and  Funding. 

(16)  Materiel  Fielding  Planning. 

(17)  Environmental. _ 


Figure  10-3.  ILS  Assessment  Considerations. 


ILS  DOCUMENTATION 

a.  Documentation. 

(1)  Associated  Support  Items  of  Equipment  (ASIOE) .  ASIDE 
are  not  normally  a  part  of  the  developed  system  however  under 
total  package  fielding  concepts  these  items  do  become  part  of 
the  complete  system  to  be  tested.  They  are  usually  listed  in 
the  technical  manuals  but  are  authorized  by  the  common  tables 
of  allowance  (CTA) ,  Table  of  Organization  and  Equipment 
(TOEs) ,  or  Joint  Tables  of  Allowance  (JTA) .  ASIOE  lists 
contain  a  description  and  the  authorized  quantity  of  the 
materiel. 

(2)  Authorized  stockage  list.  See  (13)  below. 

(3)  Basic  issue  items.  BII  are  authorized  for  issue  as  a 
component  of  the  item.  Spares  and  repair  parts  are  not^ 
normally  included  as  BII.  BII  are  those  minimum  essential 
items,  common  and  special  tools  and  TMDE,  repair  parts, 
publications,  first  aid,  and  safety-elated  equipment  required 
by  the  operator  or  crew  to — 

(a)  Place  the  end  item  in  an  operational  status. 

(b)  Cause  an  end  item  to  function  as  intended. 

(4)  Basis  of  issue  plan.  The  tentative  BOIP  (AR  71-2), 
prepared  by  the  CBTDEV,  is  submitted  with  the  requirements 
document  and  predicts  the  number  of  new  systems,  associated 
items  of  support,  and  personnel  required  in  a  unit  as  a  result 
of  the  new  or  modified  equipment.  It  identifies  items  to  be 
included  in  the  TOE,  CTA,  JTA,  and  tables  of  distribution  and 
allowance  (TDA) .  The  final  BOIP  is  submitted  32  months  before 
the  FUED. 

(5)  Critical  task  listing.  The  critical  task  listing  (AR 
611-201,  AR  611-1,  AR  71-2)  is  the  CBTDEV  and  MATDEV's  list  of 
critical  operator,  maintenance,  transportation,  supply,  and 
other  support  tasks. 

(6)  Expendable  supplies  and  materiel  (ESM)  list.  ESM  are  the 
items  needed  to  operate  and  maintain  the  end  item.  The  ESM 
list  gives  a  description,  unit  of  measure,  and  lowest  level  of 
maintenance  that  requires  the  item. 

(7)  Integrated  logistics  support  plan  (AR  700-127,  DA  PAM 
700-55) . 


Figure  10-4.  ILS  Documentation. 
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(8)  Logistics  Intelligence  File  (LIF) .  This  file  is  located 
at  Presidio,  CA,  and  contains  information  on  order  and  ship 
times  for  systems,  items,  subsystems,  components,  and 
assemblies.  It  is  a  source  of  data  for  ALDT  estimates. 

(9)  Logistics  support  analysis  (MIL-STD-1388-1A  and  2A,  AR 
700-127,  para  113-7a) . 

(10)  Lubrication  orders  (LOs) .  The  LOs  prescribe  cleaning 
and  lubrication  procedures,  proper  materiel  for  lubrication, 
lubrication  intervals,  and  locations  of  fittings  and  oil 

I  holes. 

(11)  Maintenance  Allocation  Chart.  The  MAC  reflects  a 
materiel  system's  maintenance  plan  and  is  the  overall  guide  to 
the  selection  and  allocation  of  maintenance  functions,  spares, 
repair  parts,  tools,  and  test  eguipment  to  various  maintenance 
levels.  The  MAC  identifies  the  specific  maintenance  functions 
(e.g. ,  inspect,  replace,  repair,)  that  each  maintenance  level 
is  authorized  to  perform.  It  also  establishes  a  time  standard 
for  each  maintenance  function  authorized  on  a  functional  group 
entry.  The  tools  and  test  equipment  required  to  perform  the 
maintenance  function  are  also  identified  in  the  MAC.  The  MAC 
is  published  in  technical  manuals  containing  organizational  or 
Aviation  Unit  Maintenance  (AVUM)  instructions. 

(12)  Operational  Requirements  Document  (ORD) . 

(13)  Prescribed  load  list,  authorized  stockage  list, 
essential  repair  parts  stockage  list  (ERPSL) ,  Mandatory  Parts 
List  (AR  710-2).  The  PLL  and  ASL  may  appear  in  separate 
listings.  A  repairable  exchange  is  usually  established  as  part 
of  the  ASL.  The  PLL  (organizational)  and  the  ASL 
(intermediate  and  direct  support)  give — 

(a)  Range  and  quantity  of  spares  and  repair  parts  to  be 
stocked . 

(b)  Expected  item  usage. 

(14)  Preventive  maintenance  checks  and  services  (PMCS) . 
PMCS  are  the  crew  or  operator  checks  and  services  before, 
after,  or  during  operation  that  are  prescribed  in  the  operator 
manuals  (-10  series)  to  help  retain  the  system  in,  or  restore 
it  to,  operable  condition.  PMCS  at  levels  above  crew  or 
operator  are  found  in  the  higher  level  technical  manuals. 
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(15)  Qualitative  and  quantitative  personnel  requirements 
information  (QQPRI)  (AR  71-2).  The  initial  QQPRI,  drafted  by 
the  MATDEV  and  submitted  by  the  CBTDEV,  is  a  compilation  of 
specified  Doctrine  and  Organization (D&O) ,  training,  and 
personnel  information  on  new  or  modified  materiel  systems. 

This  information  is  used  to  establish  or  revise  military^ 
occupational  specialties  (MOS)  or  additional  skill  identifiers 
(ASI)  and  to  plan  how  the  required  numbers  of  trained 
personnel  will  be  provided  to  operate  and  support  the  system. 
The  initial  QQPRI  is  prepared  and  submitted  with  the  BOIP 
feeder  data  at  the  time  the  materiel  requirements  document  is 
prepared,  and  it  is  verified  during  full-scale  development. 

The  QQPRI  lists: 

(a)  Personnel  duties  and  tasks. 

(b)  Work  units. 

(c)  Performance  standards. 

(d)  Manpower  authorization  factors. 

(e)  Recommended  MOS,  ASI,  skill  levels,  and 
organization. 

(f)  Recommended  personnel  skills  within  current, 
revised,  or  new  MOSs. 

(g)  Predictions  of  direct  productive  annual  maintenance 
manhours. 

(h)  Number  of  direct  operators. 

(i)  Quantity  of  systems  to  be  delivered  by  fiscal  year. 

(j)  Descriptive  listing  of  duty  positions  required  for 
operation  and  support. 

(k)  Suggested  MOS  from  which  personnel  can  be  obtained 
for  new  or  revised  MOS. 

(l)  Knowledge,  skills,  and  implications  for  future 
personnel  and  training. 

(16)  Repair  parts  and  special  tools  list  (RPSTL) .  The  RPSTL 
illustrates  and  lists  spares  and  repair  parts,  special  tools 
and  test  equipment,  and  other  special  support  equipment  that 
is  required  to  maintain  an  item.  The  RPSTL  supports  the  MAC 
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and  lists  special  tools  and  TMDE,  special  support  equipment, 
spares,  and  repair  parts.  The  RPSTL  is  published  as  a 
separate  manual  or  as  part  of  a  maintenance  manual  that  may 
cover  one  or  more  levels  of  maintenance. 

(17)  Support  packages. 

(a)  System  Support  Package-  Materiel  and  logistics  support 
that  has  been  established  by  the  Materiel  Developer's 
Logistics  Manager  that  is  required  to  assure  the  system  can  be 
sustained  in  the  field.  This  support  package  forms  the  basis 
for  determining  whether  the  system  can  be  sustained  by  the 
maintainers  and  receives  rigorous  operational  test  and 
evaluation. 

(b)  Test  Support  Package-  This  is  the  bill  of  materiel 
necessary  to  complete  the  necessary  tests  to  determine 
suitability  of  the  system.  All  of  the  materiel  contained  in 
this  package  generally  is  not  tested,  since  much  of  it  will 
not  be  fielded  with  the  production  system.  However,  it  can 
provide  the  tester  with  some  vital  information. 

(18)  System  supportability  assessments.  The  independent 
logistician  is  responsible  to  assess  for  the  Department  of 
Army  Deputy  Chief  of  Staff  for  Logistics  all  systems  to 
determine  whether  the  Army's  Logistics  System  is  established 
correctly  to  ensure  that  sustainability  in  wartime  can  be 
achieved.  These  assessments  are  the  responsibility  of  AMSAA 
who  reports  directly  to  Department  of  Army  and  are  maintained 
on  one  consolidated  data  base. 

(19)  Test  program  set  (TPS)  management  plan.  Test  Program  Sets 
involve  the  hardware  and  software  necessary  to  interface  with 
automatic  test  sets  used  to  diagnose  electronic  components  by 
the  logistics  personnel  in  the  field.  Testing  of  the  TPS's 
that  are  established  by  the  TPS  management  Plan  is  essential 
to  achieve  adequate  logistics  support  for  electronic  systems 
that  require  use  of  automatic  test  equipment. 
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ILS  DEMONSTRATION  AND  TESTING. 

(1)  Contractor  testing  (AR  73-1) .  During  all  system 
development  and  before  any  UT,  contractor's  data  may  provide 
insights  such  as  form,  fit  and  function, location  and 
assessability  to  components  and  line-replaceable  units  (LRU) , 
complexity  of  the  maintenance  procedures,  requirements  for 
special  tools,  peculiar  TMDE,  peculiar  support  equipment, 
additional  facilities  and  handling  equipment.  Two  reviews  of 
contractor's  testing  and  design;  the  Preliminary  Design  Review 
and  the  Critical  Design  Review  provide  the  essential 
information  on  which  supportability  and  testability  decisions 
should  be  made. 

(2)  Force  development  test  and  experimentation  (AR  73-1).  FDTE 
is  used  to  verify  the  logistics  supportability  doctrine, 
concept,  and  organization  found  in  the  ORD  and  the  DOSP.  It 
also  provides  data  on  selected  support  elements  such  as  common 
tools  and  tool  kits,  stands,  fixtures,  and  TMDE,  and 
information  on  the  required  skills  of  the  maintenance, 
support,  transportation,  and  supply  personnel. 

(3)  Logistics  demonstration  and  maintenance  demonstration  (MD) 
(AR  700-127,  DA  PAM  700-50). 

(a)  An  LD  is  the  nondestructive  operation  (tear  down  if 
required)  and  maintenance  of  a  developmental  prototype  system 
or  end  item  and  its  maintenance-significant  support  and  test 
equipment.  It  is  performed  to  evaluate  the  achievement  of 
maintainability  goals  and  the  adequacy  of  the  SSP.  A 
maintenance  demonstration  (MIL-STD-471)  and  physical  and 
functional  configuration  audits  may  be  scheduled  to  coincide 
with  the  LD. 

(b)  The  LD  is  conducted  in  an  environment  that  duplicates,  as 
closely  as  practical,  the  expected  operational  and  maintenance 
conditions  in  the  field.  This  environment  is  to  be 
representative  of  the  working  conditions,  tools,  support 
equipment,  supply  support,  facilities,  and  equipment 
publications  that  would  be  required  and  available  during 
operational  service  use  at  the  level  of  maintenance  defined  in 
the  MAC.  This  rest  will  utilizes  target  audience  maintenance 
personnel  which  are  TRADOC  school  trained  on  the  system.  This 
demonstration  serves  to  certify  the  training  that  normally  is 
contractor  provided. 

(c)  The  LD  is  performed  sufficiently  in  advance  of  major  TT 
and  OT  to  permit  certification  and  finalization  of  the  SSP. 
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(d)  A  logistics  demonstration  is  required  to  be  conducted  on 
most  (some  ACAT  III  &  IV  programs  do  not)  acquisition  programs 
by  the  PEO/PM/MATDEV.  Normally  an  LD  will  be  conducted  prior 
to  the  production  decision.  If  the  LDA  is  not  conducted  prior 
to  the  production  decision,  the  acquisition  decision  approval 
authority  is  required  to  grant  a  waiver  for  it  to  be  conducted 
following  the  production  decision.  This  will  often  happen  in 
the  case  of  NDI  acquisitions  or  modifications  to  existing 
equipment. 

(4)  Operational  testing  (AR  73-1) . 

(5)  Sample  data  collection  (AR  750-37) . 

(6)  Technical  and  operational  test  training.  This  training 
provides  the  knowledge  necessary  to  use  the  technical 
documentation,  special  and  common  tool  kits,  TMDE,  maintenance 
and  support  equipment,  and  other  ILS  functions  applicable  to 
the  system  planned  for  test.  Test  players  are  trained  using 
the  designated  support  materiel.  If  the  designated  support 
materiel  is  not  available  and  training  is  impacted,  the 
independent  operational  evaluator  will  assess  the  impact  in 
coordination  with  the  MATDEV,  who  develops  alternative 
training  to  insure  that  a  proper  test  can  be  performed. 

(7)  Technical  testing.  TT  provides  system-level  data  on  all 
the  ILS  elements  addressed  in  AR  700-127.  The  data  developed 
during  technical  testing  normally  provides  data  about  the 
system  that  is  needed  by  the  operational  tester  to  develop  a 
test  that  assures  the  system  is  suitable  for  the  eventual 
user.  It  is  extremely  important  for  the  developer  of  the  OTP 
to  participate  in  the  acquisition  meetings  before  and  during 
the  technical  testing  to  assure  all  system  peculiar 
requirements  applicable  to  the  user  are  known  and  tested  to 
meet  the  COIC. 
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Figure  10-6.  ILS  Considerations  in  the  concept  exploration 
and  demonstration  and  validation  phases. 

This  section  is  written  for  the  independent  operational 
evaluator.  The  OT&E  process  for  logistics  supportability 
between  MSs  I  and  II  is  illustrated  in  figure  10-1. 

a.  Integrated  logistics  support  plan. 

(1)  At  MS  I,  the  MATDEV  initiates  an  ILSP  (DA  PAM  700-55). 

The  ILSP  describes  the  planned  ILS  program  and  addresses  the 
total  integrated  logistics  support  system  that  includes  all 
the  ILS  elements  and  assessment  considerations.  Section  2.b  of 
the  ILSP  contains  the  logistics  supportability  T&E  concept, 
objectives,  scope,  and  ILS  issues. 

(2)  The  ILS  assessment  considerations  are  generated  by  the  LSA 
process  and  are  typically  qualitative  statements  contained  in 
the  requirements  documentation  such  as  the  MNS  and  the  ORD. 
They  are  developed  in  coordination  with  the  CBTDEV, 
logistician  (see  para  10-3) ,  and  the  independent  operational 
and  technical  evaluators  and  are  updated  at  each  MS. 

(3)  Review  the  T&E  section  of  the  ILSP  to  ensure  that  critical 
OT&E  issues  are  identified  and  resources  are  available  for 
test.  Include  these  issues  in  the  TEMP  and  ensure  that  they 
are  covered  in  the  operational  TEP. 

b.  Identifying  critical  logistics  issues. 

(1)  Identification  of  requirements  imposed  on  the  logistics 
system  by  the  design  (and  vice  versa)  are  critical  to  insuring 
O&S  costs  established  in  the  requirements  documentation  are 
met.  These  requirements  and  in  some  cases  constraints  are  best 
determined  through  attention  to  the  following  acquisition 
processes. 

(a)  Review  draft  requirements  documentation  and  attend  ORD 
joint  work  groups  to  include  RRRWG. (AR  702-3) 

(b)  Become  a  full  time  member  of  the  ILSMT  and  review  LSA 
processes. 

(c)  Become  a  member  of  the  Preliminary  Design  Review  Team  and 
the  Critical  Design  Review  Team.  These  acquisition  program 
reviews  are  considered  extremely  important  because  the 
configuration  of  the  equipment  that  will  be  tested  and 
evaluated  is  established. 
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(2)  Some  examples  of  ILS  issues  that  may  be  unrealistic: 

(a)  ••No  increase  in  maintenance  personnel •'  may  be  an 
inappropriate  issue  if  a  new  system  will  not  replace  an 
existing  system  and  there  is  no  planned  reduction  in  current 
maintenance  workload. 

(b)  ••The  system  is  to  be  supportable  by  DS  maintenance^^ ,  the 
level  of  organic  support  is  decided  by  the  LSA  process  and 
normally  not  predetermined  at  the  beginning  of  the 
acquisition. " 

(c)  ••The  system  is  not  to  require  unique  test  or  support 
equipment. " 

(d)  ••The  system  is  to  be  transportable  by  existing  rail  and 
air  inventory  assets.” 

(3)  Examples  of  typical  ILS  issues  listed  above  that  might 
place  constraints  on  the  logistical  support  system  being 
developed  must  jointly  be  addressed  by  members  of  the 
acquisition  community  in  early  assessments  especially  by  the 
designated  independent  logistician  and  the  operational  test 
evaluator.  The  early  assessments  will  address  as  a  minimum 
address  the  considerations  listed  below  as  well  as  others  that 
might  be  raised  by  any  member  of  the  acquisition  team. 

(a)  Are  the  constraints  realistic  and  appropriate  for  this 
type  of  system? 

(b)  Do  the  constraints  reflect  current  logistics  doctrine? 

(c)  Will  the  constraints  unnecessarily  limit  the  system  design 
and/or  the  logistics  design? 

(d)  Will  the  constraints  unnecessarily  preclude  the 
application  of  state-of-the-art  advances  in  system  hardware, 
software,  and  support  concepts? 

(e)  Will  constraints  unnecessarily  limit  effectiveness  and 
suitability? 

(f)  Will  the  planned  logistics  support  concepts,  doctrine,  and 
organization  conform  to  the  Army  Standard  Logistics  System? 

(AR  750-1  ,  AR  710-2) 

c.  Mission  Needs  Statement. 
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(1)  The  MNS  identifies  the  types  of  operational  unit  and  its 
basic  organizational  structure  that  will  operate  and  support 
the  system.  It  will  establish  general  guidelines  for  materiel 
support  concepts,  readiness  objectives  and  begins  the  ILS 
process . 

(2)  The  MNS  review  is  required  to  identify  inconsistencies 
between  current  or  planned  logistics  support  doctrine  and 
organization,  and  the  Army  Logistics  System  in  place  for  the 
type  system  being  developed. 

d.  Early  supportability  testing. 

(1)  Plan  EUTE  to  uncover  potential  logistics  supportability 
problems.  Effective  EUTE  uncovers  deficiencies  such  as 
components  located  in  areas  inaccessible  to  maintenance 
personnel,  requirement  for  unique  or  additional  test 
equipment,  nonstandard  components  of  any  type,  and  establish 
the  baseline  for  the  logistics  demonstration  to  be  conducted 
prior  to  lOTE. 

(2)  FDTE  is  beneficial  in  the  test  process  to  provide 
essential  data  on  logistics  support  doctrine,  concepts,  and 
organization. 

e.  Early  modeling.  Determine  whether  any  logistics  model  can 
be  effectively  utilized  during  early  evaluation  and  testing. 
If  one  is  deemed  to  be  useful  the  following  should  be 
identified  concerning  use  of  models. 

(1)  Type  of  model  such  as  the  logistics  level  to  be  analyzed. 

(2)  The  model's  characteristics. 

(3)  The  source  to  build  the  model  (or  the  existing  candidate 
model) . 

(4)  The  organization  to  run  and  maintain  the  model. 

(5)  The  resources  needed  to  support  the  application. 
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Figure  10-7.  ILS  considerations  in  the  full-scale  development 
phase. 

a.  Logistics  Support  Analysis  (LSA/LSAR) process. 

(1)  Following  MS  II  with  the  finalization  of  the  requirements 
documentation,  LSA  becomes  important  with  LSAR  generated 
during  this  phase.  LSA  considers  design  trade  off,  ILS  trade 
off  analysis,  allocation  of  tasks,  MANPRINT  Validation, 
selection  and  allocation  of  Repair  Parts,  and  generally 
validates  all  the  supportability  disciplines  established  by 
the  ILS  process. 

(2)  Some  of  the  generic  analysis  used  to  justify  and  document 
the  logistics  support  requirements  are  listed  below.  The 
specific  tasks  and  analysis  requirements  are  discussed  in 
MIL-STD-1388-1A  and  MIL-STD-1388-2A.  The  Materiel  Developer 
has  the  responsibility  to  ensure  contractor  compliance  with 
LSA  requirements. 

(a)  Have  logistics  supportability  constraints  and  alternative 
support  concepts  been  incorporated  into  the  LSA? 

(b)  Have  LSA  tradeoff,  risk,  and  sensitivity  been  analyzed 
effectively? 

(c)  Have  logistics  supportability  goals  and  objectives  been 
set,  justified,  and  reflected  in  the  system  design? 

(d)  Do  the  RAM  and  LSA  processes  support  each  other? 

(e)  Is  there  a  feedback  loop  among  the  system  design,  the  T&E 
processes,  and  the  LSA  processes? 

(f)  Have  logistics  support  concept,  organization,  and  doctrine 
been  developed? 

(3)  All  members  of  the  acquisition  community  should  provide 
feedback  to  the  LSA  process  by  reporting  uncorrected  logistics 
supportability  deficiencies  to  the  MATDEV.  As  LSA  data  becomes 
available,  the  CBTDEV  updates: 

(a)  The  BOIP. 

(b)  The  QQPRI. 

(c)  The  logistics  doctrine,  concept,  and  organization. 
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b.  Operational  Requirements  Document  (ORD) . 

1)  The  ORD  contains  an  assessment  of  the  development  of 

(a)  Quantitative  logistics  support  parameters. 

(b)  Baseline  logistics  support  concepts. 

(c)  Potential  logistics  problem  areas. 

(d)  Preferred  limits  on  the  need  for  logistics  support  element 
resources. 

(e)  Current  and  projected  changes  to  pertinent  logistics 
systems  and  procedures. 

(f)  Environment  that  the  logistics  is  required  to  operate  in. 

2)  ORD  and  its  logistics  parameters  are  updated  at  each 
milestone  decision. 

c.  Operation  of  the  Integrated  Logistics  Support  Management 
Team  (ILSMT)  and  the  Test  Integration  Working  Group (TIWG). 

1)  The  ILSMT  and  the  TIWG  are  two  of  the  principal  acquisition 
teams  that  are  used  to  ensure  supportability  is  adequately 
tested  during  the  acquisition  process  and  the  appropriate 
supportability  is  on  hand  at  IOC.  Close  coordination  is 
required  between  the  two  acquisition  disciplines  to  insure: 

(a)  LSA  reviews  are  conducted  as  required  and  LSAR 
requirements  are  delivered  by  the  contractor. 

(b)  That  all  logistics  issues  are  identified  with  resolutions 
established  and  corrective  actions (FIX  ,TEST,  FIX)  are  taken 
prior  to  IOC  to  ensure  appropriate  sustainability  exists. 

(c)  That  test  and  evaluation  is  scheduled,  properly  resourced, 
and  funded  to  investigate  the  planned  logistics  support  for 
all  systems  being  acquired  to  support  Army  operations. 

Detailed  methodology  to  support  the  T&E  process  is  quantified 
and  documented  in  the  ILSP,  Contractor's  ILS  SOW,  and  the 
TEMP. 

(d)  Assist  the  Materiel  Developer's  ILS  manager  and  T&E 
manager  with  planning,  scheduling,  and  providing  timely  US 
Government  inputs  into  the  acquisition  process. 
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,d.  Request  for  proposal. 

1)  Ensure  that  the  ILSMT  and  TIWG  directions  are  clearly 
stated  so  that  the  contractor  can  establish  for  the  US 
Government  an  organic  logistics  support  system  at  IOC  that 
does  not  rely  on  contractor  logistics  support. 

2)  Ensure  that  all  ILS  assessment  elements  with  necessary 
clarification  have  been  included  to  enable  the  contractor  to 
prepare  a  response  that  will  provide  a  suitable  logistics 
support  system. 

3)  Establish  parameters  in  the  ILS  Statement  of  Work  (SOW) 
that  will  allow  the  contractor  to  use  all  available  design 
options;  current  and  known  advances  in  logistics  engineering 
to  develop  a  logistics  support  system  that  will  guarantee  O&S 
costs  as  established  by  the  acquisition  strategy  that  will 
also  provide  a  support  system  suitable  for  the  target 
audience. 

e.  Logistics  modeling. 

1)  If  any  logistics  model  is  to  be  used  in  the  evaluation 
process  or  to  support  operational  testing,  ensure  that 
contractor,  TT,  and  OT  logistics  supportability  data  are  made 
available  to  the  organization  running  the  model. 

2)  Organization  operating  the  model  will  be  provided  the  level 
of  analysis  and  sensitivity  excursions  required  to  enhanced 
the  lOTE. 

f.  Issues  and  criteria. 

1)  Logistics  supportability  issues  and  criteria  will  be 
developed  from  the  logistics  baseline  established  in  the  MNS 
and  ORD  by  the  CBTDEV  in  coordination  with  MATDEV,  TT,  OT,  the 
Independent  Logistician,  and  other  members  of  the  acquisition 
team  as  required. 

2)  COIC's  relating  to  sustainability  will  be  refined, 
justified,  and  completed  normally  by  TIWG  and  ILSMT  actions 
and  incorporated  in  the  ILSP  and  TEMP.  The  SOW  provided  to  the 
contractor  will  contain  sufficient  details  to  assure  that  his 
design  supports  the  COICs. 

3)  If  the  logistics  supportability  issues  provided  by  the 

CBTDEV  and  the  Independent  Logistician  are  not  adequate  to 
support  the  anticipated  evaluation  or  are  without  quantitative 
criteria,  the  independent  operational  evaluator  will _ 
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logistics  supportability.  It  defines  critical  operational 
logistics  supportability  issues,  MOPs,  DRs,  and  associated 
evaluation  and  analysis  planning  necessary  to  ensure  that  the 
issues  are  adequately  answered  before  each  MS  decision.  The 
TEP  will  contain  plans  to  evaluate  all  ILS  elements  and  and 
special  logistics  requirements  identified  in  the  ILSP  and 
TEMP. 
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Figure  10-8.  ILS  Considerations  in  the  Production  and 
Deployment  Phase. 

a.  After  MS  III,  the  MATDEV  updates  the  ILSP  and  the 
production  contractors  statement  of  work  to  ensure  all 
supportability  deficiencies  are  corrected.  The  OEIs  are 
revised  in  conjunction  with  the  independent  operational  and 
technical  evaluators  to  include  the  independent  logistician 
and  other  ILS  program  participants  as  required  by  the  changes 
made  that  may  impact  their  activities. 

b.  Ensure  that  remaining  OEIs  or  deficiencies  associated  with 
the  logistics  supportability  concepts,  doctrine,  organization, 
and  materiel  are  incorporated  into  the  ILSP  and  that  the  TEMP 
is  updated. 

c.  In  the  TEMP,  include  resources  required  for  FOTE. 

d.  In  the  TEP,  identify  data  sources  and  criteria  needed  to 
complete  the  logistics  supportability  evaluation.  Address 
elements  of  the  SSP  that  required  a  waiver  or  were  not  fully 
tested  or  evaluated  during  lOTE  and  any  remaining  issues 
regarding  logistics  supportability  doctrine,  concepts,  or 
organization.  If  the  evaluation  is  to  be  supported  by  SDC  (AR 
750-37) ,  provide  input  to  the  SDC  plan. 

e.  The  following  actions  are  required  for  tracking  and 
reporting  deficiencies: 

1)  Report  deficiencies  to  the  MATDEV. 

2)  Establish  a  plan  for  validating  and  verifying  corrective 
actions. 

3)  Establish  specific  methodology  to  monitor  status  of  all 
corrective  actions. 

4)  Validate  and  Verify  completed  corrective  actions  for  all 
logistics  support  deficiencies  regardless  of  source. 

5)  Input  OT-identified  deficiencies  to  the  ATEDB. 

6)  Review  SDC  plans  to  ensure  follow-up  data  collection  and 
reporting  supports  the  requirement  to  valid  and  verify  that 
logistics  deficiencies  are  correctly  in  a  timely  cost 
effective  manner. 


Figure  10-8.  ILS  considerations  in  the  production  and  deployment 
phase. 
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7)  Use  the  Independent  Logistician's  ILSMIS  and  the 
USALEA/HQDA  DCSLOG  Command  Logistics  Review  Team  results  as 
additional  sources  of  deficiency  evaluation.  (OPTEC/AMSAA/HQDA 
DCSLOG/ AMC  MOU  and  ARll-1) . 


Figure  10-8.  ILS  considerations  in  the  production  and  deployment 
phase  (cont'd). 


ILS  ORGANIZATIONS 


(1)  The  US  Army  Materiel  Systems  Analysis  Activity  (AMSAA) , 
Army  Materiel  Command (AMC) .  AMSAA  is  the  Army's  designated 
"Independent  Logistician"  responsible  to  Department  of  the 
Army  Deputy  Chief  of  Staff  for  Logisitics  for  performing  the 
ILS  program  surveillance,  independent  logisitics 
supportability  assessments,  and  evaluations  on  all  materiel 
acquisition  programs  and  deployed  systems  with  the  exception 
of  medical  items  for  which  the  U.S.  Army  Medical  Materiel 
Agency  is  responsible.  AMSAA  will  use  data  from  both  technical 
and  operational  testing  and  include  any  other  testing  as  input 
to  its  supportability  assessments. 

(2)  Test  and  Evaluation  Command  and  Army  Materiel  Systems 
Analysis  Activity.  TECOM  and  AMSAA  are  responsible  for 
Technical  T&E  of  the  ILS  elements.  This  T&E  provides  for  the 
operational  tester  a  safe  and  usable  system  to  conduct  a 
satisfactory  user  test.  All  ILS  elements  receive  a  technical 
verification  and  certification  that  the  system  meets  the 
specifications  used  by  the  contractor  to  develop  the  system. 

(3)  Combined  Arms  Support  Command  (CASCOM)  develops  tactical 
logistics  doctrine,  concepts,  organizations,  and  materiel  for 
all  organic  levels  of  maintenance  excepting  Depot  Level  which 
is  the  responsibility  of  AMC.  LOGCEN  and  AMC  work  in  close 
coordination  to  insure  the  planned  depot  level  maintenance 
will  provide  the  necessary  support  for  the  other  organic 
support  units . 

(4)  Combat  Developer.  This  is  the  organization  element, 
normally  the  TRADOC  schools  and  centers,  that  develop  the  MNS 
and  ORD  that  serve  as  the  basis  for  any  new  materiel  system. 
The  logistics  supportability  required  is  generally  described 
in  both  the  MNS  &  ORD  through  close  coordination  with  the 
designated  Independent  Logistician  (AMSAA) .  It  is  important 
that  logistics  supportability  be  reviewed  carefully  as  early 
as  possible  in  the  acquisition  cycle  because  O&S  costs 
normally  make  up  a  very  large  percentage  of  the  total  program 
costs.  Failure  to  establish  the  proper  logistics  procedures 
and  parameters  at  program  initiation  will  result  in  higher 
support  costs  and  impact  both  technical  and  operational 
testing.  The  combat  developer  also  is  responsible  for  the 
establishment  of  the  institutional  training  processes 
necessary  to  train  both  operators  and  maintainers. 


Figure  10-9.  ILS  Organizations. 
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(5)  Materiel  Developer.  This  is  the  organizational  element 
that  converts  the  combat  developer's  requirements  into  a 
specification  that  is  furnished  to  the  industrial  base  for 
contractors  to  compete  against  to  develop  a  system  that  will 
fulfill  the  user's  needs.  The  Materiel  Developer  establishes  a 
statement  of  work  for  the  ILS  elements  using  the  logistics 
concepts  established  by  the  MNS  and  the  ORD.  It  is  critical 
for  both  the  technical  and  operational  tester  to  participate 
with  the  Materiel  Developer  during  the  processes  leading  to 
releasing  of  the  ILS  statement  of  work  to  the  respective 
contractors.  This  will  ensure  adequate  testing  of 
supportability  is  contained  is  all  test  plans. 

(6)  OPTEC.  Tester  and  evaluator,  with  the  influence  and 

guidance  of  the  OPTEC  ILS  Division,  measures  the  suitability 
of  the  supportability  concept  agreed  to  by  the  CBTDEV.  This 
concept  is  translated  into  a  COIC  followed  by  AOICs. _ _ 


Figure  10-9.  ILS  Organizations  (cont'd) 


Figure  10-10.  Example  of  a  time-line  analysis  of 

administrative  and  logistics  downtime. 

Supply  support 
PLL/ASL 

Demand  For  Item(s) 

Task  Requiring  Item(s) 

Identification  of  Item(s)  [FGC,  LCN] 
Serviceability  Code  of  Item(s) 

Direct  Exchange  of  Item(s) 

Quantity  of  Item(s)  Required 

Date  and  Time  of  Requisition  of  Item(s) 

Operating  Units  on  Item(s) 

Disposition  of  Replaced  Item(s) 
Administrative  and  Logistics  Downtime 
Item  on  PLL/ASL  Listing 
Level  of  Maintenance  Requiring  Item(s) 

Common  or  System  Peculiar  Identification 
Mission  Criticality 


Figure  10-10.  Example  of  a  time-line  analysis  of  administrative 
and  logistics  downtime. 
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Figure  10-11.  Data  requirements  for  logistics  elements. 


POL 

POL  Demand  by  System  or  Support  Vehicle 
Quantity  of  POL  Required 
Source  of  POL 

Location  of  POL 

Administrative  and  Logistics  Downtime 
Other 

Demand  For  Bulk  Items 
Demand  for  On  Board  Spares 
Demand  for  Basic  Issue  Item(s) 

Demand  for  Additional  Authorized  List  (AAL)  Item(s) 
Task  Requiring  Item(s) 

Quantity  of  Item(s)  Required 
Location  of  Item(s) 

Administrative  and  Logistics  Downtime 

Manpower  and  personnel 

Operator  and  crew 

Identification  Of  Task 
Critical  and  Non-critical  Task 
Clock  Hours  by  MOS 
Results  of  Action 
Identification  of  MOS 

Operation,  Maintenance,  and  Support  Personnel  Required 
Man  Hours  by  MOS 

Supply  Support  Personnel 

Task  Requiring  Supply  or  Support  Personnel 

Result  of  Action 

Man  Hours  for  Task  by  MOS 

Administrative  And  Logistics  Downtime 

Critical  and  Non-critical  Tasks 

MOS  of  Supply  and  Support  Personnel 

Number  of  Supply  and  Support  Personnel  Required 

Man  Hours  by  MOS 

Maintenance  Personnel 


Figure  10-11.  Data  requirements  for  logistics  elements. 


Task  Requiring  Maintenance  Personnel 
Level (s)  of  Maintenance  for  Task 
Man  Hours  by  MOS 
Clock  Hours  by  MOS 

Critical  and  Non  Critical  Task  for  Each  MOS 


Set  Up  Time  by  MOS  for  Each  Task 
Clean  Up  Time  by  MOS  for  Each  Task 
Calibration  Team  Required  for  Task 
Supply  Support  Required  for  Each  Task 
Results  of  Action 

Administrative  and  Logistics  Downtime 

MOS  of  Maintenance  Personnel 

Level (s)  of  Maintenance  for  Task  In  Manual 

Number  by  MOS  Maintenance  Personnel  Required 

Diagnostic  Time  by  MOS  for  Each  Task 

Contact  Team  Required  for  Each  Task 

TMDE  Required  for  Task 

On  and  Off  Repair 

Time  For  Task  in  Manual  (Maintenance  Allocation  Chart 
(MAC)) 

TMDE 

Test  program  sets 

Demand  for  TPS 
Identification  of  TPS 
Identification  of  ATE  Using  TPS 
Location  of  TPS 
TPS  Setup  Time 

TPS  Diagnostic  Time  (Isolation,  Location) 

Task  Requiring  TPS 

Administrative  and  Logistics  Downtime 

Maintenance  Level  Requiring  TPS 

MOS  Using  TPS 

TPS  Initialization  Time 

TPS  Utilization  Units 

Result  of  TPS  Action 

BITE 

Fault  Indication 
Fault  Isolated  Correctly 
Used  in  Maintenance 


Figure  lO-ll.  Data  requirements  for  logistics  elements  (cont'd) . 


^  Fault  Detected  Correctly 

False  Alarm 

'  Type  of  BITE 

Manual  test  equipment  and  ATE 

Demand  for  ATE 

Identification  of  ATE 

ATE  Setup  Time 

ATE  Diagnostic  Time 

Maintenance  Level  Requiring  ATE 

Time  to  failure  of  ATE 

Time  to  Calibrate  ATE 

MOS  Repairing  and  Calibrating  ATE 

Administrative  and  Logistics  Downtime 

Common  or  Special  Purpose 

Task  Requiring  ATE 

Utilization  Units  of  ATE 

ATE  of  Initialization  Time 

MOS  Operating  ATE 

Quantity  of  ATE  Required 

Time  to  Repair  ATE 

Demand  for  ATE  Calibration 

Results  of  ATE  Action 

ATE  Documentation  Required 

Tools,  tool  kits,  and  other  maintenance  equipment 

Demand  for  Item(s) 

Identification  of  Item(s) 

MOS  Requiring  Item(s) 

Utilization  Units  of  .Item 

Level  of  Maintenance  Requiring  Item(s) 

Maintenance  Clock  Hours  Used 
Task  Requiring  Item 
Location  of  Item(s) 

Identification  as  Common  or  Special  Purpose  Quantity 
Required 

Administrative  and  Logistics  Downtime 
Technical  documentation 

Manuals 

Demand  for  Manual (s) 

Task  Requiring  Manual (s) 

Identification  of  Manual (s) 

Location  of  Manual (s) 


Figure  10-11.  Data  requirements  for  logistics  elements  (cont'd). 


Level  of  Maintenance  Using  Manual (s) 

MOS  Using  Manual (s) 

Manual (s)  Procedure (s)  Used 
Error (s)  in  Manual (s) 

Result  of  Manual  Use 

Administrative  and  Logistics  Downtime 
Calibration 

Demand  for  Calibration 

Identification  of  Calibration  Documentation 

Location  of  Documentation 

Result  of  Calibration 

Tasks  Requiring  Calibration 

Calibration  Procedure (s)  Used 

Item  Requiring  Calibration 

MOS  Calibrating  System 

Administrative  and  Logistics  Downtime 

Other 

Demand  for  Other  Technical  Documentation 
Other  Documentation  Procedures  Used 
MOS  Using  Other  Documentation 
Level  of  Maintenance  Using  Manual (s) 

Task  Requiring  Other  Documentation 
Identification  of  Other  Documentation 
Location  of  Other  Documentation 
Result  of  Other  Documentation  Use 
Administrative  and  Logistics  Downtime 

Other  support  equipment 

Demand  for  Support  Item(s) 

Identification  of  Support  Item(s) 

MOS  Operating  or  Using  Support  Item(s) 

Quantity  of  Support  Item(s)  Required 
Results  of  Action 

Administration  and  Logistics  Downtime 
Common  or  special-purpose 
Task  Requiring  Support  Item(s) 

Location  of  Support  Item(s) 

Utilization  Units  of  Support  Item(s) 

Level  of  Maintenance  Requiring  Support  Item(s) 
Compatibility  of  Support-Item (s) 


Figure  10-11.  Data  requirements  for  logistics  elements  (cont'd). 
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Figure  10-12.  Example  of  a  logistics  burden  analysis  for  fuel 
consumption. 

1.  POL  consumption  rate  per  system  =  10  gallons  per  hour 
(test) . 

2.  Mission  length  *  24  hours  per  day  (OMS/MP) . 

3.  Number  of  generators  in  unit  =  20  (BOIP) . 

4.  Requirement  =  5  gallons  per  hour  per  system  (ROC). 

5.  Fuel  tanker  capacity  =  600  gallons  (TOE  and  tanker 
description) . 

6.  Unit  demand  analysis. 

a.  10  gallons  per  hour  X  24  hours  per  day  =  240  gallons  per 
day  per  system. 

b.  240  gallons  per  day  per  system  X  20  systems  =  4800  gallons 
per  day  per  unit. 

c.  4800  gallons  per  day  per  unit/ 600  gallons  per  tanker  =  8 
tanker  loads  per  day. 

d.  Requirement  was  for  four  tanker  loads  per  day.  Further 
analysis  is  required. 

e.  Tanker  turnaround  time  =  4  hours  (test) . 

f.  Number  of  units  supported  =  10  (O&O  Plan). 

g.  Storage  bladder  capacity  =  10,000  gallons  (TOE). 

h.  Manpower  per  tanker  =  4  operators  per  truck;  2  crews  per  24 
hours  (TOE) . 

7 .  Number  of  tankers  required  . 

a.  8  tanker  loads  per  day  X  10  units  supported  =  80  tanker 
loads  per  day. 

b.  Demand  =  24  hours  per  day/4-hour  turnaround  time  per  tanker 
=  6  tanker  loads  per  tanker  per  day. 

c.  80  tanker  loads  per  day  required/ 6  tanker  loads  per  tanker 
per  day  =13.3  tankers  required. 

d.  13.3  tankers  required  X  1.2  operational  availability  factor 
(.2  is  the  additional  requirement  because  of  .80  availability) 
=  16  tankers  required  in  unit  (3  are  DX  supply  items  in  unit). 

8.  Number  of  bladders  required. 

a.  4,800  gallons  per  day  per  unit  X  10  units  =  48,000  gallons 
per  day. 

b.  48,000  gallon  per  day  per  unit  supplied  x  20-day  supply 
required  =  960,000  gallons  storage  required. 

c.  960,000  gallons  storage  required/10, 000-gallon  bladder 
capacity  =  96  bladders  required. 

9.  Manpower  required. 

a.  13  tankers  in  operation  X  4  operators  per  tanker  =  52 
tanker  personnel  in  operation. 


Figure  10-12.  Example  of  a  logistics  burden  analysis  for  fuel 
consumption. 


b.  96  storage  bladders/10  bladders  per  3  personnel  X  3 
personnel  =29  personnel  in  bladder  operations. 

c.  Total  manpower  =81  persons  required. 

10.  The  additional  resources  required  are — 

a.  8  tankers. 

b.  48  bladders. 

c.  40  personnel. 


Figure  10-12 .  Example  of  a  logistics  burden  analysis 
for  fuel  consumption  (cont'd). 

Note:  If  these  resources  are  not  added,  only  half  the  total  fleet 
will  be  maintained  committable  with  fuel  or  will  be  able  to 
operate  12  of  the  24  hours  required.  Although  Ao  does  not  include 
measurements  of  fuel  availability,  this  lack  of  fuel  carries  the 
same  impact  on  Ao  as  maintenance.  Because  only  50  percent  of  the 
fuel  reserves  will  be  available,  the  Ao  for  this  generator  is,  in 
effect,  50  percent  even  before  maintenance  is  considered.  The 
logistics  burden  associated  with  this  required  increase  must  be 
identified.  In  this  fuel  consumption  analysis  example,  the 
requirement  for  twice  the  fuel  translates  into  double  the 
programmed  tankers,  bladders,  and  manpower  support  at  the  resupply 
point.  Without  redesign  or  adding  these  needed  assets,  a  50  percent 
reduction  in  operational  readiness  will  result. 
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1.  Requirement  =  move  ASL  in  one  move  (O&O  plan) . 

2.  ASL  current  volume  capacity  =  10,000  cubic  feet  (TOE  and 
supply  van  description) . 

3.  Additional  capacity  required  =  2,800  cubic  feet  (LD  and 
test) . 

4.  Additional  capacity  planned  =  1,000  cubic  feet  (BOIP  and 
supply  van  description) . 

5.  Shortfall  =  1,800  cubic  feet. 

6.  The  ASL  will  require  two  moves  or  additional  supply  vans  to 
transport  the  ASL. 


Figure  10-13.  Example  of  a  logistics  burden  analysis  for 
authorized  stockage  list  volume. 
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1.  Density  =  6,000  items  (BOIP) . 

2.  Mission  length  =  12  hours  (MP) . 

3.  Usage  rate  =  2,400  hours  per  year  (OMS) . 

4.  Reliability  requirement  =  500  MTBF  (ROC)  =4.8  failures  per 
year 

5.  Manhours  per  repair  =  2.0  hours  (test) 

6.  Reliability  estimate  =  175  MTBF  (test)  =  13.7  failures  per 
year. 

7.  Increase  in  failures  per  year  =  8.9. 

8.  Additional  failures  per  year  =  6,000  items  X  8.9  failures  = 

53.400  failures  per  year. 

9.  53,400  failures  per  year  x  2.0  manhours  per  failure  = 

166.400 

manhours  per  year. 

10.  One  manyear  =  4,456  manhours  =  37  additional  repairmen. 

Note:  Further  analysis  could  be  performed  to  determine  the 

level (s)  of  maintenance  and  MOS  at  which  the  shortage 
occurred.  In  addition,  this  shortfall  may  indicate  the  need 
for  additional  tools,  TMDE,  or  vehicles.  Because  the 
reliability  requirement  is  not  met,  the  reliability 
degradation  translates  into  approximately  three  times  as  many 
failures  as  projected.  Accordingly,  if  no  reliability 
improvement  is  obtained,  43  more  maintenance  personnel  will  be 
r equ ired  to  support  the  system. _ 


Figure  10-14.  Example  of  a  Logistics  Burden  Analysis  for 
Maintenance  Manpower. 
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1.  Density  =  6,000  items  (BOIP) . 

2.  Mission  length  =  12  hours  (MP) . 

3.  Usage  rate  =  2,400  hours  per  year  (OMS) . 

4.  Reliability  requirement  =  500  MTBF  (ROC)  =  4.8  failures  per 
year 

5.  Manhours  per  repair  =2.0  hours  (test) 

6.  Reliability  estimate  =  175  MTBF  (test)  =  13.7  failures  per 
year. 

7.  Increase  in  failures  per  year  =  8.9. 

8.  Additional  failures  per  year  =  6,000  items  X  8.9  failures  = 

53.400  failures  per  year. 

9.  53,400  failures  per  year  x  2.0  manhours  per  failure  = 

166.400 

manhours  per  year. 

10.  One  manyear  =  4,456  manhours  =  37  additional  repairmen. 

Note:  Further  analysis  could  be  performed  to  determine  the 

level (s)  of  maintenance  and  MOS  at  which  the  shortage 
occurred.  In  addition,  this  shortfall  may  indicate  the  need 
for  additional  tools,  TMDE,  or  vehicles.  Because  the 
reliability  requirement  is  not  met,  the  reliability 
degradation  translates  into  approximately  three  times  as  many 
failures  as  projected.  Accordingly,  if  no  reliability 
improvement  is  obtained,  43  more  maintenance  personnel  will  be 
required  to  support  the  system. 


Figure  10-15.  Example  of  a  Logistics  Burden  Analysis  for 
Maintenance  Manpower. 


1.  Number  of  systems  to  be  supported  in  the  unit  =  200  (TOE). 

2.  Number  of  systems  supported  in  test  =  4  (test). 

3.  Daily  utilization  per  day  =  2  hrs  per  day  (test). 

4.  Ao  for  the  system  =  .90  (test) . 

5.  Ao  for  TMDE  =  .80  (test). 

6.  Number  of  TMDE  in  test  =  1  (test) . 

7.  Number  of  TMDE  planned  for  unit  =  4  (TOE) . 

8.  Average  utilization  per  system: 

a.  2  hrs  per  day/4  systems  tested  =  .50  hrs/day  per  system. 

b.  200  systems  in  unit  X  .90  availability  *  180  systems 
operational  per  day  per  unit. 

c.  180  systems  operational  X  .50  hrs  per  day  per  system  §  90 
hrs  per  day  TMDE  utilization. 

9.  TMDE  required: 

a.  24  hrs  per  day  X  .80  TMDE  availability  =  19.2  hrs 
operation  available  per  day. 

b.  90  hrs  per  day  TMDE  utilization/ 19 . 2  hrs  available  per  day 
=4.7  TMDE  items  required. 

10.  Impact  of  fielding  with  4  TMDE:  without  combat  losses 
being  considered,  4  TMDE  can  support  the  following  number  of 
systems: 

a.  4  TMDE  authorized  X  .80  Ao  =  3.2  TMDE  available. 

b.  3.2  TMDE  available  X  24  hrs  per  day  =  76.8  hrs  operation 
per  day. 

c.  76.8  hrs  operation  per  day/. 50  hrs  TMDE  demand  per  day  per 
system  =  153.6  systems  supported. 

d.  With  180  systems  operational  per  day,  a  backlog  of  systems 
will  develop  at  the  TMDE  stations  because  only  an  average  of 
3.2  TMDE  items  are  operationai  at  any  time.  System  Ao  will  be 
degraded . 

e.  Considering  combat  losses: 

(1)  Expected  percent  lost  per  day  =  .20. 

(2)  Therefore  at  the  end  of  the  first  combat  day,  180  systems 
X  .80  =  144  systems  plus  20  in  maintenance  =  164  systems  in 
fleet  for  second  day  with  147  available  (164  X  .90). 

Note:  After  the  first  day  of  battle,  the  TMDE  will  be  able  to 
support  the  unit.  Consequently,  4  TMDE  may  be  acceptable.  The 
TMDE  analysis  example  demonstrates  how  the  logistics  burden  is 
affected  by  the  difference  between  the  force  slice  tested  and 
the  force  slice  to  be  supported.  Although  no  problem  with 
queuing  at  the  TMDE  was  observed  in  test,  considering  the 
force  slice  to  be  fielded  shows  that  a  queuing  problem  will 
result.  The  analysis  also  suggests  how  consideration  of  combat 
losses  can  affect  the  analysis  conclusions;  although  analysis 
may  show  at  first  that  the  queuing  problem,  the  a  location  of 
TMDE  may  prove  correct  when  combat  losses  are  considered. 
Expected  system  losses  in  the  first  3  days  of  battle  could 
overcome  the  queuing  problem. 


Figure  10-16.  Example  of  a  logistics  burden  analysis  for  test 
measurement  and  diagnostic  equipment. 


\c>- 


1.  Tool  utilization  =  40  maintenance  clockhours  per  month  per 
system  (OT) . 

2.  Number  of  systems  in  unit  =  60  (BOIP) . 

3.  Number  of  tools  allocated  to  unit  =  4  tools  (BOIP). 

4.  Required  tool  hours  =  40  X  60  =  2,400  maintenance 
clockhours  per  month. 

5.  Available  hours  =  720  hours  per  month  X  4  tools  =  2,800 
hours. 

6.  A  sufficient  number  of  tools  has  been  provided. 

Figure  10-17.  Example  of  a  logistics  burden  analysis  for  tools. 
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1.  Number  of  critical  operator  tasks  validated  by  military 
operators  =42  (OT) . 

2.  Total  critical  operator  tasks  technically  validated  in 
operator's  manual  =  210  (review  of  manual  and  TT) . 

3.  Percent  critical  operator  tasks  demonstrated  as  correct  by 
military  operators  =  42/210  =  20  percent. 

4.  Therefore,  80  percent  of  validated  critical  operator  tasks 
have  not  been  demonstrated  by  military  operators. 


Figure  10-18.  Example  of  a  logistics  burden  analysis  for  technical 
documentation . 
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1.  Estimated  LRU  cost  =  $50,000  (contractor  engineering 
estimate  in  current  dollars) . 

2.  Estimated  cost  to  repair  LRU  =  $1,500  (comptroller  data  in 
current  dollars) . 

3.  Operating  time  =  6,000  hours  (test). 

4.  Number  of  LRU  failures  =  6  (test). 

5.  System  operating  time  per  year  =  4,000  hours  (OMS/MP) . 

6.  Expected  number  of  failures  per  year  per  system  =  4. 

7.  Number  of  systems  to  be  fielded  =  1,000  (BOIP) . 

8.  Percent  of  LRU  repaired  =  .66  (test). 

9.  Percent  LRU  replaced  =  .33  (test). 

10.  4  failures  per  year  per  system  X  1000  systems  =  4,000 
failures  per  year. 

11.  4,000  failures  X  .66  =  2,640  repaired  per  year. 

12.  4,000  failures  X  .33  =  1,320  replaced. 

13.  2,640  failures  X  $1,500  .  $3,960,000  for  repair. 

14.  1,320  failures  X  $50,000  =  $66,000,000  for  replacement. 

15.  Total  cost  per  year  for  repair  and  replacement  = 
$69,960,000. 


Figure  10-19.  Example  of  a  logistics  cost  analysis 
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Figure  10-20.  Logistics  supportability  operational  issues  and  data  source  matrix. 


Deputy  test  director  (DTD)  for  logistics. 


Substantial  efforts  are  required  for  all  systems  to  assure  the 
planned  logistics  support  is  effective  because  adequate 
logistics/sustainability  is  always  a  critical  issue.  A  DTD  for 
logistics  is  included  in  the  OTP  resources  for  the  OT.  The  DTD 
for  logistics,  who  is  familiar  with  the  Army's  logistics 
support  system,  doctrine,  organization,  and  concepts,  has  the 
following  functions; 

a.  Assisting  in  preparing  the  Detailed  Test  Plan  (DTP) . 

b.  Collecting  logistics  supportability  data. 

c.  Controlling  the  SSP. 

d.  Assuring  realism  in  the  play  of  the  SSP. 

e.  Controlling  play  of  logistics  support  doctrine, 
organization,  and  concepts  in  test. 

f.  Determining  when  contractor  support  will  be  played. 

g.  Determining  that  the  repair  parts  and  spares  are  tracked, 
utilized  correctly,  and  disposed  of  within  the  logistics 
support  concept  stated  for  the  system. 

h.  Determining  the  validity  of  the  maintenance  and  logistic 
data  bases. 

i.  Determining  the  requirement  for  post  test  maintenance  and 
support  exercises. 


Figure  10-21.  Deputy  test  director  for  logistics. 


Table  lO-l.  Examples  of  measures  of  performance  and  baseline 
criteria  for  selected  logistics  supportability  elements 


Measure  of 
performance 


Description 


Baseline 

criterion 

(%) 


Demand  satisfaction  index  ^  ^  ^  ^ 

Percent  of  valid  systeiu—unicjue  and  inission^critical  supply 
reguests  satisfied  from  the  PLL/ASL  (i.e.^  the  item  is  in 
stock) 

80 

Demand  accommodation  index 

Percent  of  valid  system-unique  and  mission-critical  supply 
requests  accommodated  from  the  PLL/ASL  (i.e.,  the  item  is 
normally  stocked) 

80 

Mobility  index 

Percent  of  PLL/ASL  lines  that  are  transportable  in  a  single 
move.  100^ 

Volume  accommodation  index 

Percent  of  required  supply  support  space  and  volume  that  can 
be  accommodated  with  planned  storage  facilities 
100 

POL  index 

Ratio  of  the  POL  replenishment  rate  to  the  POL  consumption  or 
delivery  rate 
100 

Zero  balance  index 

Percent  of  zero  balance  PLL/ASL  lines 
10 

Repairable  exchange  index 

Percent  of  valid  repairable  exchange  requests  filled 
80 

PLL/ASL  line  item  increase  index 
Percent  of  new  PLL/ASL  line  items 
10 


Table  10-1.  Examples  of  measures  of  performance  and  baseline 
criteria  for  selected  logistics  supportability  elements. 
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Measures  of  performance  and  baseline  criteria  for  selected 
logistics  supportabilitv  elements-Continued 


Measure  of  Description  Baseline 

performance  criterion 

(%) 


Logistic  reliability  measures 

Mean  time  between  removals,  cost  per  repair,  and  others^ 
Consumption  and  replenishment  rates 

Rates  at  which  supply  support  is  consumed  and  replenished^ 
Tools,  tool  kits,  and  other  maintenance  equipment 
Tool  index 

Percent  increase  in  required  common  or  special  tools  or  tool 
kits 
0.0 

Tool  utilization  rate 

Percent  of  common  or  special  tools  or  tool  kits  utilized 
None 

Tool  hour  utilization  rate 

Ratio  of  tool  utilization  hours  to  the  total  number  of 
maintenance  clock  hours  during  the  same  time  period 
None 

Technical  documentation 

Percent  critical  tasks  or  procedures  validated 
Ratio  of  the  total  number  of  tasks  or  procedures  in  the  manual 
to  the  number  validated 
100^ 

Percent  of  erroneous  procedures  or  tasks 
Ratio  of  erroneous  tasks  and  procedures  in  the  manual 
demonstrated  to  the  total  tasks  and  procedures  demonstrated 
0® 

MAC  times 

Percent  of  MAC  times  demonstrated  by  the  soldier  to  total 
number  of  MAC  times  in  the  manual 
100^ 


Table  10-1.  Examples  of  measures  of  performance  and  baseline 
criteria  for  selected  logistics  supportability  elements  (cont'd) . 


lO 


Percent  of  NSNs  assigned  in  parts  manual 
Percentage  of  parts  that  have  been  assigned  NSNs 
100* 

No^gs • 

1.  If  mobility  of  the  PLL/ASL  is  critical  to  the  system. 

2.  There  are  no  baseline  criteria,  but  criteria  can  be 
obtained  for  individual  systems  from  the  ORD,  from  historical 
data  of  similar  systems,  or  from  the  LSAR. 


3.  There  is  no  baseline  criterion,  although  there  may  be 
requirements  in  the  ORD  or  historical  data  and  cost  estimates 
on  similar  systems. 

4.  The  ratio  should  approach  100  percent  as  the  system  nears 
the  production  decision. 

5.  The  ratio  should  approach  0.0  percent  as  the  system 
matures . 


Table  10-1.  Measures  of  Performance  and  Baseline  Criteria  for 
Selected  Logistics  Supportability  Elements. 
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Chapter  1 1 

Test  and  Evaluation  of  Reliability,  Availability  and 
Maintainability  (RAM) 


Section  I 

RAM  in  the  Materiel  Acquisition  Process 


11-1.  General 

a.  This  chapter  defines  the  RAM  related  activities  of  T&E 
throughout  the  life  cycle  of  a  materiel  system.  It  discusses  the 
development  tester  and  evaluator/assessor  (covered  by  the 
umbrella  term  "evaluator”)  and  the  operational  tester  and 
evaluator  roles  regarding  RAM  in  the  combat  and  materiel 
developments  process,  T&E  planning,  T&E  conduct,  and  T&E 
assessment. 

b.  The  material  in  this  chapter  is  to  be  tailored  for  each 
program  based  on  the  level  of  complexity  of  the  system,  the 
acquisition  phase,  acquisition  strategy,  and  the  impact  of  RAM  on 
the  performance  and  suitability  of  the  system.  As  presented,  it 
illustrates  comprehensive  application  to  the  most  complex 
systems,  but  is  intended  for  selective  application  as 
appropriate.  Army  RAM  policy  is  established  in  7^  702-3. 

11-2.  Reliability 

Table  11-1  provides  the  classic  definition  of  reliability  and 
defines  the  two  general  types  of  reliability  which  need  to  be 
addressed  in  operational  evaluation.  Mission  reliability 
addresses  the  system  effectiveness,  while  logistics  oriented 
reliability  addresses  the  burden  of  owning  and  operating  the 
system. 

(Insert  TABLE  11-1) 

a.  The  development  of  a  reliability  requirement  usually 
assumes  that  the  failure  rate  of  the  mature  system  will  be 
constant  over  a  long  period.  This  assumption  allows  the 
requirement  to  be  expressed,  not  as  a  probability,  but  as  an 
easily  measurable  parameter  directly  related  to  reliability. 

b.  In  test  and  evaluation  the  mission  reliability  parameter 
is  normally  one  of  the  following: 

(1)  Mean  Time  Between  Operational  Mission  Failures  (MTBOMF) . 

(2)  Mean  Time  Between  Failures  (MTBF) . 
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(3)  Mean  Time  Between  Mission  Aborts  (MTBMA) . 

(4)  Mean  Time  Between  Mission  Affecting  Failures  (MTBMAF) . 

c.  If  the  system  has  another  measure  of  usage  besides  time, 
then  the  parameter  is  expressed  with  those  units  such  as  miles, 
rounds,  or  events  between  failures.  For  single  shot  devices, 
such  as  a  missile  system,  reliability  is  expressed  as  a  ratio  of 
the  number  of  successes  to  the  number  of  total  attempts. 

11-3.  Maintainability  ^  ^  ^  • 

Maintainability  relates  to  the  ease  and  efficiency  of  performing 
both  corrective  and  scheduled  maintenance  on  a  system.  Table  11- 
2  defines  maintainability  and  the  two  most  common  parameters  used 
in  maintainability  assessment. 

(Insert  TABLE  11-2) 

11-4.  Availability  « 

Availability  is  used  to  assess  systems  which  spend  a  portion  of 
their  time  in  a  ready  status  and  at  some  undetermined  time  are 
required  to  initiate  a  mission.  A  system's  availability  is  a 
function  of  its  reliability  and  maintainability.  Table  11-3 
provides  the  definitions  of  availability  and  operational 
availability. 


(Insert  TABLE  11-3) 

11-5.  Operational  Versus  Technical  RAM 

The  concept  of  operational  RAM  differs  significantly  from 
technical  RAM. 

a.  Technical  RAM  examines  the  RAM  characteristics  based  only 
on  the  hardware  and  embedded  software  of  the  system.  It  focuses 
on  the  extent  to  which  the  system  meets  technical  RAM 
specifications  and  reflects  those  failures  for  which  the  system 
contractor  is  accountable. 

b.  Operational  RAM  considerations  for  a  system  relate  to  its 
hardware,  embedded  software,  typical  operators  and  maintainers, 
manuals,  tools.  Test,  Measurement,  and  Diagnostic  Equipment 
(TMDE) ,  support  equipment,  and  the  operational,  organizational, 
and  logistical  support  concepts.  Operational  RAM  quantifies  the 
degree  to  which  the  user  can  rely  on  required  system  functions 
and  the  burden  associated  with  keeping  those  functions  at  his 
disposal.  The  operational  RAM  assessment  cannot  be  disassociated 
from  the  operational  scenarios  in  which  the  system  must  function 
nor  from  the  support  environment  on  which  the  system  must  rely. 
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Section  II 
RAM  Overview 


11-6.  Overview  of  RAM  in  the  Life  Cycle 

a.  Operational  RAM  values  expressed  in  requirements  documents 
will  be  developed  in  coordination  with  the  development  and 
operational  independent  evaluators  (See  AR  702-3). 

b.  Development  and  operational  testing  will  be  oriented 
toward  providing  data  with  which  to  estimate  the  operational  RAM 
values  expressed  in  the  requirements  document.  Tests  will  be 
designed  to  ensure  that  statistically  adequate  estimates  of 
developmental  and  operational  RAM  values  are  provided.  See  Part 
Four  for  RAM  in  DT. 

c.  Testing  at  the  system  level  will  be  conducted  in 
accordance  with  the  operational  mode  summary  and  mission  profile 
(OMS/MP) .  This  testing  will  be  designed  to  reflect,  as  closely 
as  possible,  field  usage  to  contribute  consistency  to  testing  and 
to  facilitate  data  combination. 

d.  Data  from  testing  will  be  scored,  assessed,  and  aggregated 
in  accordance  with  the  provisions  of  AR  702-3. 

e.  Test  planning  will  be  a  joint  effort  oriented  toward 
maximizing  planned  statistical  combination  of  RAM  data.  The  test 
program  will  be  structured  so  that  subtests,  taken  as  a  whole, 
will  provide  a  statistically  sufficient  sample  size  to  address 
the  RAM  requirements. 

f.  Estimates  of  the  operational  RAM  values  resulting  from  the 
RAM  assessment  conference  will  be  presented  by  the  operational 
evaluator  to  the  milestone  decision  review  body  (See  AR  702-3) . 
The  operational  evaluator  may  also  present  independent  estimates 
of  the  operational  RAM  values  based  on  selected  RAM  data, 
appropriate  analytical  techniques,  and  projected  and/or  expected 
modifications  to  the  operational  employment  of  the  system  in  a 
field  environment. 

g.  The  development  evaluator  will  present  to  the  pre-ASARC  or 
MAISRC  or  IPR  estimates  of  the  RAM  values  based  on  seleotr.d  RAM 
data,  appropriate  analytical  techniques,  and  projected 
engineering  or  utilization  changes  to  be  made  in  the  system. 
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h.  When  final  development  and  operational  testing  RAM  values 
differ,  the  evaluators  will  advise  the  milestone  decision  review 
body  of  the  probable  cause.  Such  differences  will  be 
rationalized  by  the  review  body. 

i.  At  program  reviews,  program  managers  will  indicate  how 
their  programs  are  structured  to  attain  the  RAM  requirements  and 
will  (where  appropriate)  present  growth  curves  that  provide  a 
realistic  portrayal  of  how  the  system  RAM  will  grow  to  the 
desired  performance  requirements.  All  growth  curves  will  have 
development  independent  evaluation  coordination  and  will  be 
provided  to  the  operational  independent  evaluator. 

j.  The  contract  RAM  requirements  and  test  procedures  will  be 
coordinated  with  the  development  independent  evaluator  and 
provided  to  the  operational  independent  evaluator  and  combat 
developer  for  information.  This  requirement  will  be  accomplished 
through  coordination  of  the  request  for  proposal  (RFP) ,  or 
equivalent  contract  solicitation  document,  before  its  release. 

)c.  Scoring  and  assessment  of  the  contractor  data  will  be 
accomplished  in  accordance  with  the  procedures  of  AR  702-3  and 
Sections  III  and  V,  below. 

l.  The  independent  evaluator  is  responsible  for  analyzing 
system  RAM  characteristics,  and  evaluating  RAM  characteristics 
and  performance.  This  requires  selective  participation  in 
acquisition  events,  input  to  selected  planning  documents,  and ^ 
development  of  a  plan  to  quantify  system  RAM  characteristics  in 
terms  of  mission  objectives.  This  plan  requires  the  evaluator's 
understanding  of  and  input  to  the  definitions  of  the  operating 
and  support  environments,  the  operational  tasks  required  of  the 
system,  acceptable  levels  of  task  performance,  and  the 
relationship  of  tasks  to  mission  objectives. 

m.  Figure  11-1  outlines  the  RAM  process  as  it  relates  to  test 
and  evaluation.  The  following  paragraphs  define  the  independent 
evaluator's  level  of  participation  in,  contribution  to,  and 
expectations  from  the  outlined  events. 

(Insert  Figure  11-1) 

11-7.  RAM  Program  Plan  (AR  702-3,  MIL-STD-470,  MIL-STD  785) 

a.  Early  in  the  concept  exploration  phase  the  MATDEV  or 
program  managers  office  prepares  the  RAM  Program  Plan.  The  RAM 
Program  Plan  may  be  a  single  document  or  it  may  be  a  series  of 
plans,  one  for  reliability  (MIL-STD-785) ,  one  for  maintainability 
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(MIL-STD-470)  and  so  on.  The  plan  (or  set  of  plans)  defines  both 
the  contractor  and  government  RAM  programs.  The  RAM  Program  Plan 
is  developed  in  coordination  with  the  development  evaluator  and 
is  available  through  the  program  manager  to  the  operational 
evaluator,  logistician,  and  the  CBTDEV  for  comment. 

b.  The  RAM  Program  Plan  includes: 

(1)  Identification  of  and  schedule  for  the  program  tasks  to 
be  performed  in  order  to  meet  requirements  along  with  a  detailed 
description  of  how  each  task  will  be  performed  or  complied  with. 

(2)  The  procedures  to  evaluate  the  status  and  control  of  each 
task  and  identification  of  the  organization  with  authority  and 
responsibility  for  executing  each  task. 

(3)  Interrelationships  of  RAM  tasks  with  other  system 
oriented  tasks. 

(4)  Known  RAM  problems  to  be  solved;  their  anticipated  impact 
on  the  system's  capability  to  meet  the  requirements  and  the  plan 
to  solve  the  problems. 

(5)  RAM  milestones. 

(6)  Identification  of  key  personnel  and  management  structure 
for  the  RAM  program. 

c.  The  RAM  Program  Plan  lays  out  the  overall  plan  to  meet  RAM 
requirements  early  in  the  development  process  and  offers  the 
opportunity  for  the  independent  operational  evaluator  to  make  an 
early  impact  on  this  planning.  The  operational  evaluator  reviews 
the  tasks,  schedules  and  milestones  as  part  of  his- effort  to 
develop  an  effective  evaluation  strategy  complementary  to  the 
acquisition  strategy.  The  operational  evaluator  also  reviews  any 
identified  RAM  problem  areas  and  initiates  plans  to  monitor  their 
resolution. 

11-8.  RAM  Rationale  Report  (AR  702-3,  TRADOC/AMC  PAM  70-11) 

a.  The  purpose  of  a  RAM  Rationale  Report  is  to  develop  and 
document  the  RAM  requirements  for  a  proposed  system.  The  RAM 
Rationale  Report  is  developed  by  the  RAM  Rationale  Report  Joint 
Working  Group  (RRR  JWG) ,  with  the  combat  developer  having  the 
overall  responsibility  for  establishing  RAM  requirements.  The 
RRR  JWG  consists  of  the  combat  developer,  the  materiel  developer, 
the  independent  operational  evaluator  and  the  independent 
development  evaluator. 
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b.  Evaluator  familiarity  with  TRADOC/AMC  Pamphlet  70-11  RAM 
Rationale  Report  Handbook,  is  required  before  participating  in  a 
RRR  JWG  or  reviewing  a  RRR.  The  objective  of  the  independent 
operational  evaluator  participation  in  the  development  of  the  RAM 
Rationale  Report  is  to  insure  that; 

(1)  The  Operational  Mode  Summary/Mission  Profile  (OMS/MP) 
adequately  describes  the  mix  of  ways  the  system  will  be  used  in 
its  peacetime  and  wartime  operational  roles.  The  mission 
profiles  adequately  identify  the  tasks,  events,  durations, 
operating  conditions,  and  environment  for  each  mission. 

(2)  An  operational  FD/SC  is  developed  which  provides  for 
consistent  categorization  and  classification  of  test  incidents. 

It  must  contain  clearly  defined  mission  essential  functions  with 
acceptable  levels  of  degradation  for  each  function. 

(3)  The  operational  RAM  requirements  are  based  on  both 
mission  need  and  operating  and  support  (O&S)  cost  considerations 
and  provide  a  firm  basis  for  the  operational  evaluation. 

(4)  The  justification  and  rationale  for  the  RAM  requirements 
are  based  on  methodologies  and  assumptions  supported  by  the 
independent  evaluator. 

(5)  An  acceptable  baseline  and  technical  feasibility  analysis 
has  been  done  by  the  materiel  developer. 

(6)  An  audit  trail  exists  from  operational  RAM  requirements 
to  technical  requirements. 

c.  After  the  RAM  Rationale  Report  has  been  approved  and 
signed  by  all  RRR  JWG  representatives,  the  executive  overview  is 
attached  to  the  requirements  document  as  the  RAM  appendix. 

11-9.  RAM  in  the  Test  and  Evaluation  Master  Plan  (DOD  5000. 2M, 

AR  73-1) 

a.  The  TEMP  is  the  planning  document  containing  both 
technical  and  operational  critical  issues  and  criteria,  which 
include  RAM.  The  TEMP  is  the  document  in  which  the  length  of  the 
operational  test  is  determined. 

b.  The  evaluators  ensure  that  there  is  sufficient  test  length 
to  obtain  statistically  valid  results.  The  test  length  is  often 
driven  by  reliability  considerations  and  there  are  several 
methods  for  determining  the  required  length  of  test.  System, 
dollar,  and  time  constraints  govern  which  method  is  best  for  a 
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given  application. 

c.  Two  references  outline  the  most  common  procedures  to 
determine  test  sample  size.  One  is  MIL— STD-781,  Reliability 
Design  Qualification  and  Production  Acceptance  Tests: 

Exponential  Distribution.  The  other  is  DOD  3235. 1-H,  Test  and 
Evaluation  of  System  Reliability,  Availability  and 
Maintainability,  A  Primer.  A  minimum  sample  size  of  three 
systems  each  operating  one  and  a  half  times  over  the  operational 
reliability  requirement  is  a  general  rule  of  thumb  applied  to 
determine  the  absolute  minimum  sample  required. 

d.  The  TIWG  typically  has  a  RAM  subgroup  to  advise  it  on  the 
existence  and  resolution  of  RAM  issues  involving  test  planning 
and  design,  test  criteria,  contractor  testing,  technical  testing, 
operational  testing,  and  first  article/initial  production 
testing.  The  subgroup  is  made  up  of  representatives  from  the 
materiel  developer,  combat  developer,  and  the  development  and 
operational  independent  evaluators.  It  is  also  augmented  by 
others  as  appropriate. 

e.  The  evaluators  insure  that  there  is  adequate  resolution  of 
operational  concerns  such  as  length  of  test,  quantity  of  test 
items  or  problems  with  the  implementation  of  the  OMS/MP  and 
FD/SC. 

11-10.  RAM  in  Test  and  Evaluation  Planning  (AR  73-1) 

a.  In  the  lEP,  TP,  and  TEP,  the  evaluator,  tester,  and  RAM 
analyst  plan  for  the  T&E  of  system  RAM  and  its  relation  to  the 
technical  requirements  and  the  operational  effectiveness  and 
suitability  of  the  system.  The  RAM  technical  characteristics  and 
the  RAM  critical  and  additional  operational  issue (s)  provide  the 
vehicle  for  translating  the  RAM  related  requirements  into 
criteria,  measures  of  performance,  and  data  requirements  in 
planning. 

b.  Anticipated  analysis  plans  and  techniques  are  discussed 
(e.g.,  reliability  growth,  modelling,  data  aggregation  with 
development  testing,  and  methods  of  analytically  dealing  with 
corrective  actions) . 

c.  In  OT  (if  warranted  by  the  data  requirements)  OPTEC's 
automated  RAM  data  management  system  OTERAM  should  be  planned  for 
use  to  provide  for  efficient  RAM  data  collection,  reduction  and 
management.  OTERAM  also  provides  a  proven  data  base  structure, 
an  automated  database,  and  automated  production  of  Test  Incident 
Reports  (TIRs) .  Figure  11-2  shows  the  basic  OTERAM  structure. 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


The  OTERAM  system  is  generic  and  requires  some  tailoring  to 
accommodate  the  various  types  of  systems.  TECOM  employs  similar 
RAM  data  collection  systems  for  DT. 

d.  All  tests,  whether  an  automated  RAM  data  base  is  used  or 
not,  are  required  to  develop  and  forward  TIRs  to  the  Army  central 
TIR  storage  facility.  TIR  preparation  methods,  forms  and 
procedures  for  input  to  the  data  base  are  contained  in  Chapter 
17. 


e.  The  RAM  content  of  test  design  consists  of  refinement  of 
the  RAM  data  requirements,  procedures  for  insuring  that  the  test 
scenarios  follow  the  OMS/MP,  a  description  of  the  maintenance 
environment,  initial  planning  for  RAM  data  collector  training, 
RAM  data  form  management,  data  authentication,  review  and 
validation,  development  of  TIRs,  and  refinement  of  the  RAM  data 
base  (when  applicable) . 


(Insert  Figure  11-2) 

11-11.  RAM  in  the  Detailed  Test  Plan 

The  Detailed  Test  Plan  (DTP)  is  the  document  in  which  plans  are 
made  for  the  detailed  control  of  the  maintenance  and  logistics 
during  the  test.  It  is  governed  primarily  by  the  System  Support 
Package  and  the  Doctrinal  and  Organizational  Test  Support 
Package.  The  tester  RAM  analyst  insures  that  all  RAM 
considerations  with  respect  to  data  collection,  data  processing 
and  data  reporting  have  been  adequately  addressed. 

11-12.  RAM  in  the  Conduct  of  Testing  (AR  73-1) 

a.  The  evaluator,  tester,  and  their  RAM  analysts  prepare  for 
Test  Readiness  Reviews  by  reviewing  the  following  RAM  related 
areas  to  identify  problem  areas  which  may  have  a  negative  impact 
on  the  successful  conduct  of  the  test. 

(1)  Quantity  of  test  items  and  schedules  of  delivery. 

(2)  Adequacy  and  completeness  of  System  Support  Package  (SSP) 
to  include  Test  Measurement  and  Diagnostic  Equipment  (TMDE) , 

Built  In  Test  and  Built  In  Test  Equipment  (BIT/BITE) ,  spares 
and  repair  parts,  operator  and  maintenance  technical  manuals, 
availability  of  special  and  common  tools,  and  required  support 
equipment  (trucks,  generators,  special  handling  equipment,  etc.). 

(3)  Training  of  maintainers  and  operators. 
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(4)  Data  collection  planning/preparation  to  include  RAM  data 
collection  forms,  RAM  data  collector  training  and  availability, 
RAM  data  management  and  quality  control,  and  the  quantity  and 
quality  of  RAM  data  collectors. 

(5)  Reduction  and  automation  of  RAM  data  to  include  RAM  data 
base  structure  and  programming,  quality  control  of  reduced  data, 
planning  and  programming  for  output  products,  test  Incident 
Report  (TIR)  preparation,  and  transmission  of  TIR  to  the  data 
base. 

(6)  Planning  and  procedures  for  the  RAM  Data  Authentication 
Group  (DAG) ,  if  required. 

(7)  Schedule  and  plans  for  the  RAM  scoring  and  assessment 
conferences. 

(8)  Unsettled  differences  with  the  FD/SC. 

(9)  Estimates  of  RAM  requirements  based  on  previous  testing, 
maintenance  and  logistics  demos,  and  contractor  testing  as 
appropriate. 

b.  Serious  problems  or  potential  "show  stoppers”  are  raised 
to  the  commander  as  soon  as  possible  before  the  review. 

c.  All  system  components  including  hardware,  software,  ground 
support  equipment,  training,  manuals,  and  TMDE  undergo  testing. 

d.  The  system  configuration  should  remain  fixed  during  OT  or 
at  a  minimum,  corrective  actions  should  be  implemented  in  groups. 

e.  The  tester  RAM  analyst  insures  that  all  RAM  data  is 
properly  collected,  reduced,  authenticated  and  validated.  The 
evaluator  RAM  analyst  insures  that  all  RAM  data  is  appropriately 
reported  and  analyzed.  The  tester  RAM  analyst  typically  acts  as 
chairman  of  the  RAM  subgroupr  of  the  DAG  and  evaluators  or  their 
RAM  analysts  chair  the  RAM  scoring  conference  and  the  RAM 
assessment  conference. 

f.  All  RAM  TIRs  are  provided  to  TECOM  for  inclusion  in  the 
Army's  central  TIR  storage  facility  (see  Chapter  17). 

11-13.  RAM  in  the  Test  and  Evaluation  Reporting  (AR  73-1) 

a.  The  RAM  portion  of  the  lER  or  TER  includes  (or  references 
the  data  base  containing)  all  the  RAM  data  collected  as  part  of 
the  evaluation.  In  addition  the  report  usually  includes 
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estimates  of  the  RAM  parameters  based  on  the  results  of  the 
scoring  or  assessment  conference.  RAM  estimates  can  differ  from 
those  established  in  the  assessment  conference  if  a  audit  trail 
and  rationale  are  provided. 

b.  Clear  identification  is  to  be  made  of  any  data  taken  under 
abnormal  circumstances,  which  for  any  reason  is  known  by  the  test 
directorate  to  be  atypical.  The  report  also  includes  any  test 
constraints  or  limitations  which  could  have  had  an  impact  on  the 
typicality  or  realism  of  the  RAM  data. 

c.  When  an  automated  data  base,  such  as  OTERAM,  is  used  to 
store  RAM  data  from  a  large  test,  a  description  of  the  data  base 
and  its  contents  are  provided  in  the  report. 

d.  Operational  RAM  is  a  major  factor  in  the  evaluation  of 
both  operational  effectiveness  and  suitability  and  is  therefore 
focused  on  mission  accomplishment  and  sustainability.  Assessment 
of  system  RAM  performance  is  made  with  the  analysis  and  findings 
required  to  provide  decision  makers  with  a  clear  picture  of; 

(1)  The  demonstrated  operational  RAM  estimates  as  compared  to 
their  requirements. 

(2)  The  projected  values  at  IOC  and  the  actions  required  in 
order  to  meet  these  projections. 

(3)  The  implications  and  risks  of  not  meeting  the 
requirements. 

(4)  Relevant  findings  indicative  of  ways  to  improve  the 
system  RAM. 

11-14.  RAM  Test  Extensions 

Tests  are  sometimes  extended  for  the  sole  purpose  of  collecting 
more  RAM  data  in  order  to  gain  statistical  confidence  in  meeting 
the  requirements.  If  the  extension  requires  changes  from  the 
original  plan  with  regard  to  the  system's  operating  environment, 
maintenance  or  logistics  support,  or  data  collection,  this  will 
be  done  through  a  test  change  proposal. 


Section  III 

RAM  Scoring  Conferences 


11-15.  General 

The  procedures  for  scoring  and  assessment  conferences  were 


11-10 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


developed  to  allow  a  fair  determination  of  RAM  data  bases  (AR 
702-3) .  The  procedures  provide  a  disciplined  set  of  protocols 
and  rules  for  conducting  RAM  conferences.  Any  spokesperson  or 
conference  chairperson  who  feels  the  intent  of  this  document  is 
being  misconstrued  should  immediately  report  concerns  to  the 
command  focal  point  for  RAM  who  will  assist  in  resolving  the 
problem  or  in  soliciting  resolution  of  the  matter  through 
appropriate  command/decision  authority  channels. 

11-16.  RAM  Scoring  Conference  Objectives 

Formal  scoring  conferences  are  held  during  and  immediately  after 
a  DT  and  an  OT.  The  objectives  of  the  scoring  conferences  are  to 
establish  a  common  test  data  base  and  to  insure  that  a  proper  and 
consistent  categorization  (assigning  classification  and 
chargeability)  of  test  incidents  is  made  using  the  approved 
FD/SC. 

11-17.  RAM  Scoring  Conference  Membership 

The  materiel  developer,  combat  developer,  independent  operational 
evaluator,  and  independent  developmental  evaluator/assessor  each 
designate  a  single  spokesperson  prior  to  the  initial  scoring 
conference.  Whenever  possible  the  principal  spokesperson  is  to 
be  the  same  representative  who  participated  in  developing  the 
FD/SC.  Conference  voting  members  include  the  four  principal  RAM 
spokespersons . 

a.  Personnel  representing  the  development  tester,  operational 
tester,  and  other  relevant  government  personnel  also  participate 
as  required.  The  logistician  is  invited  to  scoring  conferences 
as  an  observer  and  receives  copies  of  conference  results. 

b.  The  principal  spokespersons  will  make  up  the  decision 
making  body  of  the  scoring  conference.  They  will  perform  their 
function  within  the  guidelines  of  the  this  pamphlet  (supplemented 
by  agreed  upon  operating  procedures) . 

11-18.  Scoring  Conference  Chair 

The  chair  is  the  materiel  developer  for  DT  conferences  and  the 
operational  evaluator  for  OT  conferences  and  assessment 
conferences.  The  chair  may  be  the  voting  member  or  the  chairing 
command  may  provide  a  dedicated  chairperson.  The  chair  is 
responsible  for  maintaining  order  and  allowing  reasonable 
participation  by  all  government  participants  to  include  voting 
members  and  observers.  The  chair  will  schedule  all  scoring 
conferences  and  be  responsible  for  the  following; 

a.  Administrative  functions,  including  preparing  and 
distributing  conference  minutes. 
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b.  Ensuring  that  the  conference  is  conducted  effectively  and 
efficiently. 

c.  Carrying  out  the  procedures  established  by  guidance 
documents  (supplemented  where  necessary  by  consensus  agreements 
of  the  principal  spokespersons) . 

d.  Ensuring  system  contractor  personnel  do  not  attend  or 
directly  participate  in  RAM  Scoring  Conferences  which  address 
data  intended  to  support  evaluation/assessment  of  their  system's 
operational  RAM  parameters. 

11-19.  Pretest  Meeting 

The  principal  spokespersons  should  convene  prior  to  all  DT  or  OT 
or  at  the  first  official  scoring  conference  of  a  given  test 
phase.  The  meeting  is  usually  chaired  by  the  scoring  conference 
chair.  The  four  principal  spokespersons  and  the  developmental 
and  operational  testers  will  constitute  the  minimum  essential 
membership  of  the  pretest  meeting.  The  pretest  meeting  will  be 
conducted  to: 

a.  Review  and  establish  a  common  understanding  of  system 
requirements  and  the  failure  definition  and  scoring  criteria 
(FD/SC) ,  explanation  of  terms,  and  factors  used  in  calculating 
the  RAM  estimates  (examples  are  item  life  units  and  repair  and 
logistic  time) . 

b.  Establish  the  minimum  essential  data  requirements  for 
applying  the  approved  FD/SC  and  developing  estimates  of  RAM 
parameters. 

c.  Identify  the  parent  organizations  of  the  principal 
spokespersons . 

d.  Establish  system  specific  procedures  for  the  conduct  of 
scoring  conferences  (such  as  how  incidents  will  be  discussed,  any 
limitations  or  restrictions  on  discussion,  and  who  may 
participate  in  the  discussion) . 

e.  Establish  procedures  for  the  corrective  action  process; 
these  procedures  must  include  the  criteria  for  evaluating  the 
effectiveness  of  corrective  action. 

11-20.  Incident  Classification  and  Chargeability 

a.  The  conference  consists  of  an  organized  discussion  of  each 
of  the  test  incidents,  its  surrounding  circumstances,  cause  (when 
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known) ,  impact  on  the  operational  scenario,  maintenance  actions, 
hardware /software  conditions,  etc.  Based  on  the  FD/SC,  decisions 
are  made  by  each  principal  spokesperson  as  to  the  classification 
and  chargeability  of  each  incident. 


b.  All  decisions  are  to  be  made  in  accordance  with  previously 
established  guidelines  for  operation.  All  decisions  will  be  by 
majority  vote  of  the  principal  spokespersons  unless  there  is  no 
majority  opinion  regarding  the  classification  and  chargeability 
of  the  incident.  The  developmental  evaluator/assessor  will  make 
the  final  scoring  determination  (tie-breaking  vote)  during 
scoring  conferences  for  DT,  contractor  test,  and  production  phase 
testing.  The  operational  evaluator  will  make  the  final  scoring 
determination  (tie-breaking  vote)  during  scoring  conferences  for 
OT  and  other  user  tests. 


c.  Minority  opinions  are  written  (by  the  dissenter) 
justifying  dissenting  votes  and  are  included  in  the  scoring 
conference  minutes. 

d.  To  facilitate  the  conference,  the  tester  of ten  provides 
the  initial  categorization  or  prescore  for  each  test  incident. 

e.  All  test  incidents  are  scored  in  a  two  step  process  using 
the  approved  FD/SC.  The  first  step  is  to  classify  a  test 
incident  into  categories,  e.g.  mission  affecting  failure,  type  of 
maintenance  action,  and  non-RAM.  In  the  second  step  the  incident 
is  charged  to  the  underlying  cause  of  the  incident. 

f.  Scoring  decisions  consider  the  Design  Reference  Mission 
Profile  (DRMP)  for  the  system.  The  DRMP  is  a  predetermined 
composite  mission  profile  which  considers  the  function (s)  and 
operating  environment  of  a  system.  It  is  based  on  the  OMS/MP  for 
the  system.  It  provides  a  consistent  basis  for  system  design  and 
test,  and  provides  for  consistency  among  tests  used  to  estimate 
the  RAM  parameters. 

g.  An  incident  may  be  left  unscored  or  tabled  only  if  the 
majority  of  the  principal  spokespersons  feel  that  additional  data 
regarding  the  incident  is  necessary  to  ^  support  the  incident 
classification  and  chargeability  decision. 

h.  Incidents  previously  scored  may  be  reopened  if  a  principal 
spokesperson  can  establish  that  additional  data  on  the  incident 
has  been  gathered,  and  the  chairperson  or  a  majority  of  the 
spokespersons  agree  to  return  to  that  incident.  Even  if  the  ^ 
incident  is  not  reopened,  the  additional  data  may  be  entered  into 
the  minutes  of  the  meeting. 
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i.  Participation  by  any  observers  will  be  through  or  at  the 
request  of  the  chairperson  or  one  of  the  principal  spokespersons. 
Observers  should  not  be  "gagged"  merely  to  expedite  the  pace  of 
the  conference.  It  is  the  responsibility  of  the  chair  to  assure 
that  all  spokespersons  and  observers  with  information  or  value  to 
add  to  the  proceedings  can  be  heard. 

j .  Scoring  conferences  should  be  conducted  by  telephone  or 
correspondence,  if  feasible,  particularly  when  only  a  few 
incidents  are  to  be  considered.  A  face-to-face  conference  will 
be  held  at  the  request  of  chairman  or  any  of  the  principal 
spokespersons . 

k.  Scoring  conferences  will  be  scheduled  to  accommodate  the 
principal  spokespersons;  a  conference  requires  at  least  three 
principal  spokespersons.  If  one  of  the  principal  spokespersons 
elects  not  to  attend,  the  other  three  spokespersons  will  conduct 
the  conference  as  a  three-member  deliberation  with  majority  rule 
by  the  spokespersons  on  all  actions.  The  absent  member  will 

•  recognize  the  scoring  conference  results. 

l.  More  details  on  scoring  conference  procedures  are 
contained  in  AR  702-3. 

11-21.  Designation  of  Responsibility  for  Corrective  Action 
As  part  of  the  evaluation  of  test  incidents,  the  scoring 
conference  will  designate  responsibility  for  investigating  the 
incident,  initiating  corrective  action  as  necessary,  and 
reporting  results.  Activities  normally  responsible  for 
corrective  action  include: 

a.  The  materiel  developer  for  contractor  and  government 
furnished  equipment  (CFE  and  GFE) ,  including  CFE  and  GFE  hardware 
and  software. 

b.  The  tester  for  test  conditions  not  representative  of  the 
field  environment. 

c.  The  combat  developer  for  training  and  operational 
concepts . 

11-22.  Changes  to  FD/SC 

The  spokespersons  cannot  make  any  changes  to  the  approved  FD/SC 
which  modify  the  mission  essential  functions  or  RAM  parameters. 
Any  changes  to  the  FD/SC  affecting  these  functions  and  parameters 
must  be  formally  coordinated  and  approved  through  the  RAM 
Rationale  Report  approval  process,  and  provided  to  the  testers 
and  Army  logistician.  If  such  changes  are  made  after  start  of 
test,  all  incidents  are  scored  according  to  the  revised  FD/SC  to 
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assess  the  effect  of  the  change. 

11-23.  Distribution  of  Test  RAM  Data 

The  appropriate  test  activity  will  distribute  incident  reports 
and  necessary  maintenance  data  for  all  incidents  to  be  scored  at 
a  scoring  conference.  These  data  will  be  distributed  at  least 
two  weeks  before  the  conference  (or  as  agreed  upon  at  the  pretest 
meeting) .  If  the  data  is  unverified  or  incomplete,  the  test 
activity  will  present  to  the  scoring  conference  any  changes  to, 
or  amplification  of,  the  data. 

11-24.  Review  of  Test  RAM  Data 

For  efficiency  at  the  scoring  conference,  each  principal 
spokesperson  will,  prior  to  the  conference,  review  the  initial 
scoring  determination  for  each  incident  and  identify  areas  of 
disagreement.  If  any  spokesperson  has  not  received  the  test 
data,  the  scoring  conference  will  be  delayed  until  each 
spokesperson  has  had  sufficient  time  to  review  the  data. 

11-25.  Contractor  Participation  in  Scoring  Conferences 
The  following  is  Army  policy  regarding  system  contractor 
participation  in  test  and  evaluation  activities: 

a.  Contractors  for  the  tested  system  are  not  allowed  at  any 
scoring  conference  in  which  RAM  incidents  to  be  scored  will  be 
used  in  any  assessment  of  operational  RAM  parameters  which  may 
support  a  full  rate  production  decision. 

b.  Discussions  with  system  contractor  personnel  may  be 
necessary  to  ensure  full  technical  understanding  of  test 
incidents;  however,  discussions  with  system  contractor  personnel 
will  be  held  separately  from  the  scoring  conference.  A  formal 
written  record  will  be  kept  by  the  project  manager  of  all 
separate  government /contractor  discussions  of  test  incidents  to 
include  issues,  contractor  positions,  casual  analysis,  and  other 
pertinent  data. 

c.  If  the  materiel  developer  spokesperson  needs  access  to 
contractor  expertise  during  the  conference,  the  chair  may,  at  his 
discretion,  recess  the  scoring  conference  to  permit  materiel 
developer  consultation  with  the  contractor.  The  chair  may, 
subject  to  the  dissent  of  any  spokesperson,  axlow  the  materiel 
developer  to  provide  a  contractor  technical  presentation  on  a 
pertinent  aspect  of  the  system  to  the  conference  members  during 
the  recess.  Conference  members  may  question  the  contractor 
representatives  on  the  what,  where,  how,  and  why  of  the  incident, 
but  may  not  discuss  any  proposed  scoring  with  the  contractor 
present. 
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d.  DT  is  also  conducted  to  achieve  RAM-Durability  (RAM-p) 
maturity  and,  as  such,  can  only  occur  if  the  testing  is  designed 
to  find,  analyze,  fix,  and  verify  problems  through  testing  in  a 
timely  manner.  These  factors  suggest  that  engineering  level 
discussions  with  system  contractor  personnel  are  both  encouraged 
and  required.  These  discussions  should,  in  general,  take  place 
prior  to  or  during  the  conference;  however,  contractor  personnel 
should  NOT  be  physically  present  during  the  formal  government 
discussion  and  voting  periods. 

e.  In  developmental  test  programs  where  it  is  known  that 
testing  will  be  conducted  under  conditions  (OMS/MP,  stresses, ^ 
environmental  conditions,  test  support,  and  system  configuration) 
similar  to  OT,  and  an  operational  test  is  to  be  conducted  during 
the  same  phase,  OPTEC  will  notify  the  materiel  developer  that  the 
developmental  test  results  are  to  be  combined  with  OT  results. 

If  agreed  to  by  the  materiel  developer,  system  contractor 
participation  in  the  DT  scoring  conferences  will  be  the  same  as 
for  OT  scoring  conferences. 

11-26.  Scoring  Conference  Results 

The  chairman  will  publish  minutes  of  each  scoring  conference 
giving  results  of  scoring  and  RAM  calculations  based  on  incidents 
scored  and  usage  to  date.  These  preliminary  assessments,  based 
on  individual  scoring  conference  results,  must  be  accompanied  by 
a  statement  indicating  that  formal  evaluation  is  still  underway, 
and  that  the  preliminary  values  presented  are  not  the  final  or 
official  assessment  of  RAM  performance. 

11-27.  Final  Test  Data  Base 

At  the  final  scoring  conferences  that  address  data  to  be  used  for 
a  decision,  information  concerning  any  previously  scored  test 
results  will  be  reviewed  by  the  conference.  A  final  test  data 
base  identifying  test  length  and  test  incidents  will  be 
established. 


Section  IV 

Corrective  Action  for  RAM  Incidents 


11-28.  Corrective  Action  Efforts  (AR  702-3) 

Each  activity  will  initiate  appropriate  corrective  action  on 
chargeable  failures  and  provide  a  detailed  analysis  of  these 
incidents.  The  materiel  developer  will  take  the  lead  in  the 
analysis  of  failure  incidents,  and  will  sponsor  corrective  action 
reviews,  as  appropriate.  Corrective  actions  will  not  be 
considered  in  the  initial  classification  of  incidents. 
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Corrective  actions  can  only  be  considered  when  establishing  the 
final  test  data  base. 

11-29.  Corrective  Action  Process 

The  corrective  action  process  will  be  used  to  eliminate  or  reduce 
RAM  failure  modes  identified  during  a  test. 

a.  This  process  begins  at  the  scoring  conference  or  in  cases 
of  critical  incidents  at  the  time  of  the  incident.  A  critical 
incident  is  ohe  which  may  cause  severe  injury,  severe 
occupational  illness,  death,  or  major  system  damage. 

b.  As  part  of  the  evaluation  of  test  events,  the  scoring 
conference  designates  responsibility  for  investigating  the 
incident,  initiating  corrective  action,  and  reporting  the 
results.  Activities  responsible  for  corrective  action  include 
the  materiel  developer  for  hardware,  software,  TMDE,  support 
equipment  and  manuals,  the  tester  for  failures  caused  by  improper 
test  conditions,  and  the  combat  developer  for  failures  related  to 
training  and  operational  concepts. 

c.  Each  activity  initiates  appropriate  corrective  actions  and 
provides  a  detailed  analysis  of  these  incidents  to  the  members  of 
the  scoring  conference.  The  materiel  developer  takes  the  lead  in 
the  analysis  of  failure  incidents,  and  sponsors  corrective  action 
reviews  as  appropriate. 

d.  The  status  of  corrective  actions  will  be  provided  to  the 
scoring  conference  spokespersons  and  the  Army  logistician. 
Corrective  actions  may  then  be  considered  during  the  RAM 
Assessment  Conference. 

11-30.  Evaluation  of  Corrective  Actions 

Five  steps  will  be  used  in  evaluating  the  corrective  actions: 

a.  Failure  analysis  adequacy. 

b.  Appropriateness  of  corrective  action. 

c.  Demonstration  of  corrective  action  by  test. 

d.  Verification  of  future  implementation  of  corrective 
action. 


e.  Evaluation  of  effectiveness  of  corrective  action. 
11-31.  Reporting  Corrective  Action 

Each  activity  with  responsibility  for  corrective  action  will 
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report  on  the  corrective  actions  taken  for  each  failure  mode. 
While  the  test  is  in  progress,  the  responsible  activities  will 
provide  the  progress  made  on  the  first  three  steps  cited  above. 

A  final  assessment  of  all  five  steps  will  be  made  at  the  RAM 
Assessment  Conference. 

11-32.  Reliability  Failure  Review  Boards  (AR  702-3) 

After  the  test  a  Corrective  Action  Review  Team  (CART)  meeting  may 
be  called  by  the  materiel  developer.  The  CART  process  is  a  tool 
which  supports  the  MATDEV's  required  corrective  action  review 
process.  Its  purpose  is  to  determine  adequacy  and  effectiveness 
of  planned  and  implemented  corrective  actions.  The  CART  is  an 
extension  of  the  scoring  conference  and  may  be  composed  of  the 
same  members.  The  results  of  the  CART  are  used  by  the  MATDEV  in 
developing  estimates  of  projected  system  RAM  characteristics. 
These  estimates  or  projections  may  also  be  included  in  the 
independent  evaluation  and  compared  to  the  system's  RAM 
requirements.  If  either  evaluator  disagrees  with  the  RAM 
projections  made  by  the  CART,  he  includes  his  independent 
projections  in  the  report  and  highlights  his  disagreement  with 
the  CART  projections. 


Section  V 

RAM  Assessment  Conferences 


11-33.  General 

The  procedures  for  scoring  and  assessment  conferences  were 
developed  to  allow  a  fair  determination  of  RAM  data  bases  (AR 
702-3) .  The  procedures  provide  a  disciplined  set  of  protocols 
and  rules  for  conducting  RAM  conferences.  Any  spokesperson  or 
conference  chairperson  who  feels  the  intent  of  this  document  is 
being  misconstrued  should  immediately  report  concerns  to  the 
command  focal  point  for  RAM  who  will  assist  in  resolving  the 
problem  or  in  soliciting  resolution  of  the  matter  through 
appropriate  command /decision  authority  channels. 

11-34.  RAM  Assessment  Conference  Objectives 

A  RAM  assessment  conference  is  held  at  the  completion  of  each  OT 
or  before  a  major  decision  review.  The  objectives  of  the  RAM 
Assessment  Conference  are  to  establish  a  common  test  data  base 
and  to  determine  "demonstrated”  operational  RAM  estimates  that 
serve  as  the  baseline  for  all  independent  assessments. 

11-35.  RAM  Assessment  Conference  Membership 

The  materiel  developer,  combat  developer,  independent  operational 
evaluator,  and  independent  developmental  evaluator/assessor  each 
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designate  a  single  spokesperson  for  the  assessment  conference. 
Whenever  possible  the  principal  spokesperson  is  to  be  the  same 
representative  who  participated  in  the  scoring  conference. 
Conference  voting  members  include  the  four  principal  RAM 
spokespersons . 

a.  Personnel  representing  the  development  tester,  operational 
tester,  and  other  relevant  government  personnel  also  participate 
as  required.  The  logistician  is  invited  to  assessment 
conferences  as  an  observer  and  receives  copies  of  conference 
results. 

b.  The  principal  spokespersons  will  make  up  the  decision 
making  body  of  the  assessment  conference.  They  will  perform 
their  function  within  the  guidelines  of  the  this  pamphlet 
(supplemented  by  agreed  upon  operating  procedures) . 

11-36.  Assessment  Conference  Chair 

The  chair  is  the  operational  evaluator  for  all  assessment 
conferences.  The  chair  may  be  the  voting  member  or  the  chairing 
command  may  provide  a  dedicated  chairperson.  The  chair  is 
responsible  for  maintaining  order  and  allowing  reasonable 
participation  by  all  government  participants  to  include  voting 
members  and  observers.  The  chair  will  schedule  all  assessment 
conferences  and  be  responsible  for  the  following: 

a.  Administrative  functions,  including  preparing  and 
distributing  conference  minutes. 

b.  Ensuring  that  the  conference  is  conducted  effectively  and 
efficiently. 

c.  Carrying  out  the  procedures  established  by  guidance 
documents  (supplemented  where  necessary  by  consensus  agreements 
of  the  principal  spokespersons) . 

d.  Ensuring  system  contractor  personnel  do  not  attend  or 
directly  participate  in  RAM  Assessment  Conferences  which  address 
data  intended  to  support  evaluation/assessment  of  their  system's 
operational  RAM  parameters. 

11-37.  RAM  Assessment  Conference  Procedures 

The  RAM  assessment  conference  is  conducted  under  the  guidelines 
of  scoring  conferences,  except  that  no  tie  breaking  vote  exists. 
The  assessment  conference  reviews  the  test  profiles  and  results, 
in  conjunction  with  the  OMS/MP,  to  identify  which  test  phases  or 
configurations  are  relevant  for  use  in  determining  operational 
RAM  estimates.  Decisions  are  made  by  majority  vote  of  principal 
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spokespersons.  There  is  no  tie  breaking  vote.  In  the  event  that 
no  majority  opinion  is  reached,  then  each  agency  reports  its  own 
RAM  assessment  and  the  magnitude  and  the  basic  reasons  for  the 
differences  are  elevated  to  the  respective  headquarters  for 
necessary  action.  Unresolved  differences  are  reported  at 
decision  reviews. 

a.  Attempts  should  be  made  to  achieve  the  objectives  of  the 
RAM  assessment  conference  by  telephone  or  correspondence  whenever 
possible;  however,  a  conference  will  be  held  at  the  request  of 
any  of  the  principals. 

b.  If  there  is  a  significant  difference  of  opinion  at  the  RAM 
Assessment  Conference,  spokespersons  must  advise  their  respective 
headquarters  of  both  the  magnitude  and  the  basic  reasons  for  the 
difference.  Unresolved  differences  will  be  reported  at  a 
decision  review. 

c.  The  DRMP  will  be  used  to  review  the  test  profiles  and 
results  to  identify  test  phases  or  configurations  that  are 
relevant  for  use  in  determining  RAM  estimates.  The  test  data 
base  may  be  partitioned  for  analysis  according  to  environmental 
conditions,  stresses,  and  by  systems,  subsystems,  or  major  items. 
A  test  designed  in  accordance  with  the  DRMP  eliminates  the  need 
for  further  adjustment.  If  the  DRMP  is  not  followed,  procedures 
based  on  the  relationship  between  DRMP  and  test  profiles  will  be 
used. 

11-38.  Aggregation  of  Test  RAM  Data 

Aggregation  of  data  will  be  considered  and  will  address  all  RAM 
parameters  in  the  ORD.  The  conference  determines  if  DT  and  OT 
data  can  be  aggregated  (NOTE:  If  the  data  are  to  be  aggregated 
for  ACAT  I  and  II  systems  entering  full  rate  production,  the 
contractor  involvement  restrictions  of  PL  99-661  and  PL  100-180 
will  also  apply  to  the  DT  scoring  and  data  collection) . 

a.  When  aggregation  is  not  feasible,  both  DT  and  OT  results 
will  be  separately  presented  at  subsequent  decision  reviews  with 
an  explanation  of  significant  differences.  If  data  are  not 
aggregated,  the  principals  must  provide  a  detailed  explanation  of 
the  reasons  in  the  conference  minutes. 

b.  The  main  reason  where  DT. data  can  not  be  aggregated  with 
OT  data  is  that  DT  is  not  normally  conducted  under  operational 
conditions,  with  user  troops  and  in  conjunction  with  the  OMS/MP. 

11-39.  Adjustment  of  Scored  Data 

The  conference  will  determine  which  method  of  assessment  provides 
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the  most  accurate  representation  of  the  system 

end  of  test.  For  tests  conducted  in  accordance  with  the  OMS/MP 
and  the  DRMP  and  with  a  fixed  system  configuration,  no  adjustment 
of  the  scored  data  will  be  considered,  the  demonstrated  estimates 
are  based  on  unmodified  scoring  conference  results.  For  tests 
conducted  with  a  changing  and  evolving  system  configuration, 
either  Reliability  Growth  or  Engineering  Analysis  techniques  may 
be  used  in  the  determination  of  demonstrated  RAM  values. 

a.  Reliability  Growth  techniques  provide  an  objective  means 
for  assessing  the  impact  that  fixes  have  had  on  the  reliability 
of  the  hardware  tested.  Reliability  growth  tracking  techniques 
can  assess  demonstrated  reliability  of  a  system  and  provide  an 
objective  means  for  assessing  the  impact  that  fixes  have  on  the 
reliability  of  the  hardware  tested.  These  techniques  may  be 
applied  to  systems,  subsystems,  or  major  items.  The  analysis  may 
be  further  divided  according  to  environment  or  other  test  ^ 
conditions.  MIL-HDBK-189 ,  Reliability  Growth  Management,  is  a 
principal  source  of  statistical  methodology  for  assessing  the 
reliability  growth  aspects  of  the  Army  programs. 

b.  Engineering  analysis  techniques  make  use  of  engineering 
judgement  and  test  data  in  assessing  the  impact  of  fixes  on  the 
RAM  of  the  system  configuration  and  may  be  used  to  assess  the  ^ 
final  test  data  base.  The  principal  spokespersons  will  determine 
if  the  classification  status  of  corrected  failure  modes  has 
changed,  based  on  the  five  steps  in  paragraph  11-30,  above, 

(1)  Each  principal  spokesperson  subjectively  determines  if 
the  chargeable  status  of  corrected  failure  modes  has  changed^ 
based  on  failure  analysis  adequacy,  effectiveness  of  corrective 
action,  and  verification  of  the  fix  through  testing  after 
corrective  action  was  implemented. 

(2)  A  failure  identified  as  "not  relevant"  will  not  be  used 
for  computing  the  demonstrated  estimate  if  a  majority  of  the 
principal  spokespersons  agree  that  there  is  concrete  evidence 
that  a  failure  mode  v  11  not  recur  in  operational  environments, 
and  the  fix  does  not  create  any  new  failure  modes. 

jf  the  failure  rate  of  a  particular  mode  has  been  reduced 
to  a  lower  rate  (but  the  mode  has  not  been  eliminated) ,  the 
failure  rate  observed  after  the  change  will  be  prorated  for  the 
entire  test  length. 

(4)  Only  fixes  that  have  been  verified  effective  by  adequate 
government  testing  of  system,  subsystems,  or  components  (as 
determined  by  the  four  voting  members  of  the  assessment 
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conference)  may  be  used  to  reduce  the  number  of  relevant 
failures. 

c.  As  chairman  of  the  RAM  assessment  conference  the 
independent  operational  evaluator  insures,  using  the  above 
procedures  and  others  in  AR  702-3,  that  the  resulting  RAM 
estimates  are  realistic  and  represent  what  can  be  expected  of  the 
tested  system. 

11-40.  Contractor  Participation  in  Assessment  Conferences 
Contractor  Involvement.  System  contractor  personnel  will  not 
participate  in  RAM  assessment  conferences  which  address  data 
intended  to  support  evaluation  or  assessment  of  the  system's 
operational  RAM  parameters. 

11-41.  RAM  Assessment  Conference  Products 

The  chairman  will  publish  minutes  of  the  assessment  conference 
giving  results  of  assessment  and  RAM  calculations  based  on 
conference  results.  Products  of  the  conference  are  incorporated 
into  the  minutes  and  include: 

a.  The  common  data  base  under  which  all  assessment  for 
achievement  of  RAM  requirements  will  be  evaluated. 

b.  Demonstrated  RAM  estimates  that  serve  as  a  baseline  for 
all  independent  assessments. 


Section  VI 

RAM  Estimates,  Assessments,  and  Predictions 


11-42.  Resolution  of  Differing  Estimates 

The  proper  fora  to  identify  and  resolve  differences  are  the 
scoring  and  assessment  conferences.  Where  differences  exist 
(such  as  scoring  or  data  base  determination)  between  individual 
agencies  and  RAM  Assessment  Conference  determinations,  a  failure- 
by-failure  explanation  is  provided  in  evaluations,  assessments, 
or  test  reports.  Unresolvable  differences  will  be  elevated  to 
appropriate  headquarters  before  any  decision  review. 

11-43.  Individual  RAM  Estimates 

Independent  RAM  estimates  may  be  developed  by  the  independent 
evaluators/assessors,  materiel  developer,  and  combat  developer, 
based  on  analysis  of  the  test  data  base,  if  done  appropriately. 

a.  Any  deviations  from  the  agreed  upon  categorization  or 
demonstrated  estimates  will  be  clearly  identified  to  provide  a 


11-22 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


well  established  audit  trail. 

b.  Reports  and  decision  briefings  will  address  relationship 
of  the  independent  estimates  to  demonstrated  estimates.  The 
reports  or  briefings  will  provide  supporting  rationale  for 
independent  estimates. 

c.  Independent  estimates  may  result  from  different  incident 
assessment  criteria,  projections  based  on  corrective  actions, 
differences  in  analytical  techniques,  or  other  variations  in  data 
base  presentations.  Independent  estimates  are  valuable  in 
gaining  maximum  insight  into  RAM  data  bases  and  the  performance 
and  potential  of  the  system. 

11-44.  Failure  to  Demonstrate  RAM  Requirements 

When  the  requirements  have  not  been  demonstrated,  projected  RAM 
estimates  will  be  developed  before  the  next  acquisition  phase. 


a.  The  materiel  developer  will  develop  projected  values  using 
test  data,  engineering  judgment,  and  other  pertinent  information 
to  estimate  RAM  performance  expected  at  the  beginning  of  the  next 
phase  or  in  field  operations.  The  materiel  developer  will 
provide  projected  values  to  the  scoring  conference  participants 
and  will  define  the  rationale  used  to  develop  them. 

b.  A  projection  can  account  for  proposed  fixes  incorporated 
after  the  end  of  test,  and  for  late  fixes  that  were  incorporated 
near  the  end  of  the  test  period.  However,  the  projection  may  not 

fully  reflected  in  the  demonstrated  RAM  values  because  of 
limited  test  exposure. 

c.  The  lOE  will  report  to  the  Vice  Chief  of  Staff  of  the  Army 
an  assessment  of  the  effectiveness  of  the  incorporated  and 
proposed  fixes.  All  independent  evaluators/assessors  will  report 
predicted  RAM  values. 

11-45.  Breaching  of  RAM  Thresholds  . 

If  the  point  estimate  for  a  RAM  parameter  is  below  the  threshold, 
the  assessment  conference  will  conduct  analyses  to  determine  if  a 
threshold  breach  has  occurred.  Inputs  to  this  analysis  include; 

a.  Engineering  estimates  from  design  disciplines  (e.g. , 
thermal  analysis,  worst  case  analysis,  failure  modes  effects,  and 
criticality  analysis) . 

b.  Contractor  test  results  (e.g.,  reliability  demonstration, 
component/ subsystem  testing,  factory  screens  and  acceptance 
tests) . 
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c.  DT  point  estimates  and  confidence  intervals. 

d.  OT  point  estimates  and  confidence  intervals. 


Section  VII 

Operational  Reliability  Analysis 


11-46.  General 

In  order  to  address  the  reliability  of  a  system,  the  RAM  analyst 
must  have  a  clear  understanding  of  its  planned  operational 
environment . 

11-47.  OMS/MP 

The  specific  conditions  for  system  operation  (e.g.  mission  types, 
frequency,  and  duration(s),  required  tasks,  and  environment)  are 
established  in  the  OMS/MP.  The  OMS/MP  originates  as  an  annex  to 
the  MNS  and  is  refined  for  use  in  developing  RAM  requirements  in 
the  RRR. 

11-48.  FD/SC 

The  analyst  must  also  be  intimately  familiar  with  the  FD/SC,  also 
contained  in  the  RRR.  The  FD/SC  contains  the  agreed  upon 
guidelines  for  classification  and  chargeability  of  test  incidents 
and  lists  the  system's  mission  essential  functions.  Test 
incidents  are  classified  with  the  FD/SC  based  on  their  effect  on 
the  system's  mission  and  logistics  oriented  reliability. 

11-49.  Reliability  Classifications 

The  most  common  reliability  classifications  in  OT&E  are  listed  in 
Table  11-4. 


(Insert  Table  11-4) 

a.  Other  classifications  are  acceptable  if  justified  by  a 
peculiar  system  or  its  mission.  For  example,  another  mission 
reliability  classification  is  Mission  Aborts.  If  a  system 
performs  several  functions,  emerging  TRADOC  (CASCOM)  terminology 
reflects  the  use  of  Mission  Aborts  (MA)  and  Mission  Affecting 
Failures  (MAF) .  OMF  has  always  been  a  compromise  term  lying  to 
the  middle  of  the  spectrum  between  a  MAF  and  a  MA.  MA  can  be 
used  with  MAF  to  differentiate  between  those  failures  which  abort 
a  mission  versus  those  which  degrade  mission  effectiveness.  In 
the  hierarchy  shown  in  Figure  11-3,  MA  are  a  subset  of  MAF  which 
are  in  turn  a  subset  of  EMA. 

b.  Systems  most  likely  will  not  have  requi]^ments  for  all  the 
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reliability  classifications  listed  in  Table  11-4,  but  if 
warranted,  these  reliability  classifications  should  be  addressed 
with  an  investigative  issue (s)  in  the  lEP  or  TEP. 

c.  The  analyst  must  understand  the  relationship  between  and 
impact  of  the  various  mission  and  logistics  oriented  reliability 
classifications  selected  for  the  system.  Since  mission 
reliability  considers  only  critical  failures,  the  mission 
reliability  of  a  system  is  normally  higher  than  the  logistics 
reliability.  However,  some  systems  which  have  only  a  few  failure 
modes  may  have  no  substantial  difference  between  the  mission  and 
logistics  reliability. 

d.  Figure  11-3  illustrates  the  relationship  of  the  most 
common  reliability  classifications. 

(Insert  Figure  11-3) 

11-50.  Reliability  Redundancy 

In  order  to  assess  mission  reliability,  the  subsystem 
malfunctions  which  cause  the  loss  of  a  mission  essential  function 
and  the  redundant  features  of  the  system  must  be  clearly 
established.  One  recommended  method  is  to  include  in  the  FD/SC  a 
matrix  which  correlates  the  mission  essential  functions  to 
specific  subsystems.  An  example  this  type  of  matrix  is  provided 
in  Figure  11-4.  Use  of  this  matrix  does  not  imply  that  mission 
essential  functions  are  only  lost  through  hardware  or  software 
malfunctions.  Operational  reliability  assessment  includes 
failures  which  are  caused  by  others  (chargeable  to  accident, 
crew,  manuals,  maintenance  personnel,  support  equipment,  and 
training) . 


(Insert  Figure  13-4) 

11-51.  Probabilistic  Reliability 

Reliability  is  expressed  as  the  probability  that  a  system  will 
operate  at  a  given  level  of  functionality  under  a  specified  set 
of  conditions  for  a  specified  period  of  time.  The  given  level  of 
functionality  is  specified  in  the  FD/SC  through  identification  of 
mission  essential  functions  and  maximum  levels  of  degradation  for 
those  functions.  The  specified  set  of  conditions  is  defined  by 
the  OMS/MP.  The  specified  period  of  time  is  given  by  the  mission 
length  in  the  OMS/MP. 

11-52.  Mean-Time-Between-Fai lures  (MTBF) 

The  development  of  a  reliability  requirement  usually  assumes  that 
the  failure  rate  of  the  mature  system  will  be  constant  over  a 
long  period.  This  assumption  allows  the  reliability  requirement 


11-25 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


to  be  expressed,  not  as  a  probability,  but  as  an  easily 
measurable  parameter  directly  related  to  reliability.  For 
hardware  reliability,  this  parameter  is  the  MTBF,  or  for 
operational  reliability,  the  mean  time  between  operational 
mission  failure  (MTBOMF) . 

11-53.  Reliability  requirements 

A  primary  responsibility  of  the  operational  RAM  analyst  is  to 
assess  the  extent  to  which  the  system  meets  the  operational 
reliability  requirement.  Early  participation  in  the  RRR  Working 
Group  by  the  evaluator  or  RAM  analyst,  provides  the  opportunity 
to  impact  the  elements  which  drive  the  assessment  of  operational 
reliability  requirements. 

11-54.  Independent  Reliability  Assessment 

An  "Army”  assessment  of  reliability  is  made  at  the  assessment 
conference  using  data  scored  in  accordance  with  the  approved 
FD/SC.  The  independent  evaluator  participates  in  this  assessment 
and  includes  it  in  the  independent  evaluation  (but  is  not 
limited  by  it) .  In  his  independent  assessment,  the  evaluator 
addresses  the  many  considerations  which  are  relevant  to  system 
reliability  and  its  understanding.  The  following  topics  are  to 
be  considered  when  developing  an  independent  assessment  of 
operational  reliability* 

a.  If  substantially  different  failure  rates  occur  across  test 
phases,  scenarios  or  other  test  periods,  the  reliability  analysis 
may  need  to  be  segmented  or  analytically  developed  from  the 
separate  reliability  estimates  for  each  test  phase,  depending  on 
the  reason  for  the  difference.  This  is  a  real  concern  when 
different  test  scenarios  or  phases  use  different  system 
configurations  or  exercise  different  system  capabilities  (e.g.,  a 
live  fire  phase  and  a  dry  fire  phase).  In  addition,  it  is  a 
concern  when  different  test  scenarios  or  phases  are  conducted 
with  different  intensities  of  operation  or  under  different  test 
or  environmental  conditions. 

b.  If  there  were  any  modifications  or  configuration  changes 
made  to  the  system  during  the  conduct  of  test,  consideration  is 
to  be  given  to  the  use  of  reliability  growth  as  a  technique  for 
estimating  the  reliability. 

c.  If  there  are  any  critical  failures  which  must  be  corrected 
prior  to  fielding  the  system,  the  operational  reliability  is  to 
be  estimated  both  as  tested  and  assuming  the  failure  is 
successfully  corrected. 

d.  If  the  independent  evaluator  submitted  any  dissenting 
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opinions  on  operational  mission  failures  at  either  the  scoring  or 
assessment  conferences,  he  reports  the  reliability  as  scored  by 
the  scoring  conference  and  independently  estimates  the 
reliability  based  on  his  independent  scores. 

e.  If  there  are  pattern  failures  or  there  is  a  particular 
system  function,  test  condition,  or  failure  category  (hardware, 
crew,  software,  maintenance,  etc.)  which  caused  an  inordinately 
high  number  of  failures,  particular  attention  is  to  be  given  to 
possible  means  of  correction  through  training,  system  redesign, 
changes  to  operating  procedures,  etc.  The  objective  of  this  type 
exploration  is  to  extend  the  reliability  analysis  past  a  record 
of  what  happened  in  test,  to  an  analysis  which  provides  insight 
into  the  potential  reliability  of  the  system. 

f.  If  the  independent  evaluator  disagrees  with  the  assessment 
conference  decision  on  whether  to  aggregate  data  from  more  than 
one  test,  the  evaluator  develops  an  independent  estimate  from  his 
preferred  data  set  and  supports  it  with  a  discussion  of  why  he 
disagreed  with  the  aggregation  decision.  In  making  the 
aggregation  decision,  particular  attention  is  to  be  given  to  the 
extent  to  which  different  approaches  to  test  conduct  and 
equipment  management  might  have  impacted  failure  data. 

g.  If  a  test  consists  of  more  than  one  test  item  of  a  given 
type,  the  RAM  analyst  checks  to  insure  that  the  failure  rate  is 
consistent  across  similar  pieces  of  equipment.  Whenever  a  test 
item  or  group  of  items  is  found  to  differ  substantially  from  the 
other  equipment,  explanations  are  to  be  sought  in  terms  of  item 
age,  component  age,  abnormal  crew  stress,  abnormal  test 
intensity,  etc.  Depending  on  the  magnitude  of  the  difference 
between  the  failure  rates  and  the  number  of  items  involved, 
separate  reliability  estimates  may  be  required,  an  estimate  may 
be  required  excluding  the  atypical  items,  or  if  the  cause  can  be 
identified,  some  other  appropriate  adjustment  to  the  data  may  be 
required. 

h.  After  the  process  of  analyzing  the  failure  data,  carefully 
considering  the  above  factors  and  adjusting  the  data  where 
appropriate,  the  RAM  analyst  estimates  the  operational 
reliability  and  compares  it  to  the  requirement.  Estimates  are 
made  based  on  the  tested  configuration  as  well  as  estimates 
projected  to  IOC  assuming  planned  and  required  fixes  are 
successfully  implemented  and  demonstrated  by  testing. 

i.  Any  minority  score  used  in  the  independent  reliability 
estimate  is  to  be  clearly  identified  along  with  the  rationale 
supporting  the  minority  opinion. 
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j.  When  the  equipment  is  modified  between  test  phases  or 
between  tests  conducted  under  similar  conditions,  test  control 
procedures  and  intensities,  and  if  most  of  the  failures  are 
chargeable  to  hardware,  reliability  growth  methodology  is  to  be 
considered  in  projecting  reliability  estimates  to  the  IOC  time 
frame.  The  methodology  is  detailed  in  MIL-HDBK-189,  13  Feb  1981, 
and  is  a  powerful  tool  for  projecting  failure  rates  from  test 
data  taken  on  systems  which  have  undergone  modification. 
Additional  literature  on  the  reliability  growth  technique  is 
found  in  MIL-STD-781-D,  17  Oct  1986;  MIL-HDBK-781 ,  14  Jul  1987; 
and  MIL-HDBK-338,  15  Oct  1984. 

k.  The  RAM  analyst  also  assesses  the  combined  OMFs  and 
essential  maintenance  actions  (EMAs)  in  a  manner  similar  to  that 
discussed  above  except  that  there^are  usually  no  requirements 
against  which  to  compare  the  parameter  estimates.  He  also 
analyzes  the  combined  OMFs,  EMAs  and  unscheduled  maintenance 
actions  (UMAs)  in  a  similar  manner.  Resulting  parameters  are 
included  in  the  evaluation  and  interpreted  appropriately. 


Section  VIII 

Operational  Availability  Analysis 


11-55.  General 

The  RAM  analyst  estimates’  the  operational  availability  (Aq) 
requirement  based  on  the  wartime  OMS/MP.  Because  the  meaning  of 
the  Aq  requirement  is  dependent  on  specific  assumptions  for 
Administrative  and  Logistics  Downtime  (ALDT) ,  specific  parameters 
defined  in  the  wartime  OMS/MP  and  specific  definitions  in  the 
FD/SC,  assessment  against  the  requirement  must  be  subject  to  the 
same  assumptions. 

11-56.  Measuring  Aq  ih  OT 

Exclusive  use  of  test  data  do  not  usually  provide  a  responsible 
estimate  of  the  Aq  requirement.  The  analyst  uses  the  test  data 
to  estimate  the  components  of  Aq  making  sure  they  conform  to  the 
assumptions  of  the  requirement.  The  ratio  of  standby  time  to 
operating  time  needs  to  approximate  that  required  by  the  OMS/MP, 
and  the  ALDT  needs  to  approximate  that  assumed  in  the 
requirement. 

11-57.  Assessment  of  Aq  from  OT  Data 

When  test  data  do  not  provide  a  good  estimate  of  individual 
components,  values  assumed  in  development  of  the  requirement  are 
substituted. 
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a.  As  an  example,  the  test  estimate  of  ALDT  is  usually 
unrealistic  as  it  is  not  subject  to  the  logistic  supply  system^ 
used  in  the  field  or  to  numerous  other  factors  which  drive  up  its 
value  (e.g.,  delays  due  to  other  systems  competing  for 
maintenance  or  supply  resources).  When  this  is  the  case,  the 
ALDT  estimate  assumed  in  development  of  the  requirement  is 
substituted  for  the  test  ALDT. 

b.  If  the  test  was  not  long  enough  for  the  system  to 
experience  a  representative  sample  of  preventive  maintenance 

(PM) ,  the  amount  of  PM  per  operating  hour  assumed  when  developing 
the  requirement  is  substituted  for  that  experienced  in  test. 

c.  The  analyst  calculates  the  Aq  subject  to  his  best 
analytical  judgment  with  respect  to  the  use  of  test  data  or 
alternative  estimates  for  the  Aq  components.  The  computed  value 
is  then  subjectively  compared  to  the  wartime  Aq  requirement. 

d.  On  some  systems  it  would  also  be  important  for  the 
evaluator  to  address  the  Peacetime  Availability  because  of  system 
support  cost  and  readiness  requirements.  For  example,  the  most 
important  aspect  of  a  nuclear  missile  is  that  it  be  available  if 
required. 

11-58.  Assessing  Availability  through  Models 

a.  Often  the  operational  evaluator  is  faced  with  a  test 
limitation  stemming  from  fewer  test  items  being  available  than 
would  be  needed  to  equip  the  smallest  size  unit  which  would 
normally  fight  with  the  system.  This  is  typical  for  systems 
which  are  networks  or  systems  of  systems.  For  example,  a 
Remotely  Piloted  Vehicle  (RPV)  battery  would  be  fielded  with 
three  forward  control  stations  and  two  each  rear  control 
stations,  launchers  and  recovery  platforms.  The  RPV  lOT  was 
conducted  with  only  one  of  each  major  assemblage. 

b.  To  overcome  this  type  of  limitation,  a  model  may  be  useful 
for  representing  the  unit's  ability  to  accomplish  its  mission, 
taking  into  account  redundancy,  tactical  procedures  for  the 
entire  battery,  and  priorities  of  the  flights  requested.  The 
objective  of  such  a  model  would  be  to  use  the  RAM  measures  based 
on  the  test  finding  for  each  assemblage  type  in  order  to  estimate 
the  mission  availability  or  productivity  for  the  basic  fighting 
unit. 

11-59.  Operational  Availability  Computational  Procedures 
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a.  Aq  are  the  most  common  measures  of  System  Readiness 
Objectives  (SRO) .  Aq  may  be  defined  as  the  ratio  of  uptime  to 
total  time,  the  ratio  of  operating  time  and  standby  time  to  total 
time,  the  percentage  of  systems  required  to  be  operational  (e.g. , 
8/10  =  .80),  or  the  probability  of  being  able  to  start  a  mission. 

b.  The  components  of  A^  (i.e.,  operating  time,  uptime,  and 
standby  time)  also  have  many  definitions  and  interpretations. 

c.  Peacetime  Operational  Availability  (Aqp)  .  In  general,  the 
equation  shown  in  figure  11-5  should  be  used  for  assessing  Aop 
whose  estimates  consider  all  calendar  time,  maintenance  time,  and 
ALDT. 


(Insert  figure  11-5) 

(Insert  figure  11-6) 

d.  Wartime  Operational  Availability  (Aqw) •  Estimates  of  Aqw 
are  to  be  derived  using  the  formula  shown  in  figure  11-6.  Total 
calendar  time  is  not  used  to  calculate  (Aq^)  in  testing. 

Garrison  time,  weekends,  and  holidays  when  no  testing  is  ^  ^ 

scheduled  are  excluded.  Standby  time  is  included  only  if  it  is 
part  of  the  system's  mission  requirements  (e.g.,  "hide”  mode). 
Figure  11-7  and  11-8  recommend  data  sources  and  adjustments  to 
the  components  of  Aqp  and  Aq^. 

(Insert  figure  11-7) 

(Insert  figure  11-8) 

e.  Logistics  supportability  data  will  be  categofized  using 
the  ILS  elements  and  their  impact  on  Aqw  with  measured  adequacy 
of  the  SSP,  a  critical  item  in  the  evaluation  of  Aq. 

f .  Since  Aq  is  difficult  to  precisely  measure  in  the  test 
environment,  sensitivity  analyses  have  to  be  planned  to  indicate 
how  changes  in  the  logistics  supportability  measures  affect 
availability.  Figure  11-9  gives  an  example  of  one  way  to  show 
this  sensitivity. 


(Insert  figure  11-9) 


Section  IX 

Maintainability  Analysis 
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11-60.  Maintenance  Measures 

Maintenance  times  include  diagnostic  time,  active  maintenance 
time  (replace,  adjust  or  repair)  and  test  or  checkout  time. 
Logistics  or  administrative  delays  in  maintenance  are  not 
included.  Analysis  of  these  maintenance  times  is  typically 
focused  on  two  primary  operational  parameters,  the  mean  time  to 
repair  (MTTR)  and  the  maintenance  ratio  (MR) .  MTTR  is  the 
average  clock  time  to  perform  active  corrective  maintenance.  MR 
is  the  manhours  of  maintenance  expended  per  hour  of  operation. 
MTTR  and  MR  are  normally  calculated  for  each  maintenance  level 
for  the  system  as  well  as  a  total  MTTR  and  MR  for  the  overall 
system.  Estimates  of  these  parameters  (or  others  when  addressed 
in  the  requirements  document)  are  compared  to  the  operational 
maintainability  requirements  and  augmented  by  an  analysis  of 
their  implications  on  the  units  capability  to  provide  the 
requisite  maintenance  support. 

11-61.  Maintenance  Calculations 
Analysis  of  maintainability  data  include: 

a.  Maintenance  times  from  test  are  individually  compared  to 
Maintenance  Allocation  Chart  (MAC)  estimates.  If  observed  times 
differ  excessively  from  the  MAC  estimates,  they  are  highlighted 
as  they  indicate  potential  problem  areas. 

b.  Maintenance  times  which  are  exceptionally  long  or  short 
compared  to  other  times  for  the  same  maintenance  action,  are 
considered  for  analytic  treatment  as  outliers  or,  if  the 
circumstance  can  be  determined  to  be  atypical,  for  elimination 
from  the  estimate  against  the  requirement.  The  best  statistical 
treatment  of  maintenance  data  with  outliers  is  to  use  a  Lognormal 
(Pareto)  Distribution  to  compute  the  measures  of  central  tendency 
and  dispersion  for  the  data  set.  The  Lognormal  Distribution  is 
especially  applicable  to  MTTR. 

c.  The  distribution  of  maintenance  times  is  to  be  checked  in 
order  to  uncover  any  excessive  maintenance  actions  which  would 
significantly  drive  up  the  maintenance  parameter  estimates.  If 
excessive  times  are  found,  they  are  to  be  highlighted  and  ways  to 
reduce  them  are  to  be  explored  (see  Lognormal  Distribution, 
above) .  When  appropriate,  the  maintenance  parameters  can  then  be 
calculated  both  as  tested  and  as  projected  based  on  any  proposed 
fix. 


d.  The  proportion  of  individual  maintenance  actions 
attributable  to  diagnostics  is  investigated  and  when  found  to  be 
large,  highlighted  as  a  problem  area.  Also  when  the  required 
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BIT,  BITE  and  TMDE  have  not  been  made  available  for  test,  an 
analysis  is  performed  of  the  potential  reductions  in  diagnostic 
time  which  can  reasonably  be  expected  when  the  BIT,  BITE  and  TMDE 
are  fielded.  Maintenance  parameters  are  to  be  expressed  as 
tested  and  as  projected  based  on  this  of  type  analysis. 

e.  The  distribution  of  maintenance  workload  over  maintenance 
echelons  is  expressed  in  terms  of  a  Maintenance  Burden  (MB) .  It 
is  used  to  estimate  the  impact  on  each  maintenance  echelon  of  the 
workload  imposed  by  the  new  system  when  considered  in  combination 
with  the  workload  imposed  by  existing  systems. 

f.  Maintenance  times  which  were  affected  by  test  peculiar 
circumstances  are  to  be  singled  out  and  excluded  in  the  estimates 
of  the  required  maintenance  parameters.  This  includes 
circumstances  where  the  maintenance  procedures  used  were 
substantially  different  from  those  prescribed  in  the  manuals, 
were  executed  by  personnel  other  than  prescribed  to  perform  the 
maintenance  action,  were  supported  by  equipment  not  normally 
authorized  at  the  prescribed  maintenance  echelon,  or  were 
executed  by  maintenance  personnel  solely  dedicated  to  the  test. 

g.  When  there  is  enough  data  on  a  particular  maintenance 
action  to  show  a  distinct  learning  trend,  use  the  data  associated 
with  the  mature  maintenance  capability  for  maintainability 
analysis. 
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Figure  11  -2.  OTERAM. 
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Figure  11-4.  Mission  Reliability  Matrix 
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NOTES : 


(1)  Must  have  capability  to  sustain  a  minimum  of  30  mph 
for  mission  duration. 

(2)  Required  for  night  missions  only. 

(3)  Only  one  UHF  or  VHF  radio  is  required  to  be 
operational  for  a  mission. 


Figure  11-4. 
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Aop= 


TOT  +  TST 


TOT  +  TST  +  TCM  +  TPM  +  TALDT 


Aop  -  Peacetime  operational  availability 
TOT  -  Total  calendar  operating  time 
TST  -  Total  calendar  standby  time 

TCM  -  Total  chargeable  corrective  maintenance  time  during 
calendar  time 

TPM  -  Total  chargeable  preventive  maintenance  time  during 
calendar  time 

TALDT  -  Total  chargeable  administrative  and  logistics  downtime 
_ during  calendar  time  _  _ 


Figure  11-5.  Equation  for  peacetime  operational 
availability. 


AOW= 


TOT  +  TST 


TOT  +  TST  +  TCM  +  TPM  +  TALDT 


Aow  -  Wartime  operational  availability 

TOT  -  Total  operating  time  during  mission  scenario 

TST  -  Total  mission  standby  time 

TCM  -  Total  chargeable  mission  critical  corrective  maintenance 
time 

TPM  -  Total  chargeable  mission  critical  preventive  maintenance 

time  .  .  , 

TALDT  -  Total  chargeable  mission  critical  administrative  and 

logistics  downtime 


Figure  11-6.  Equation  for  wartime  operational  availability. 
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Wartime  Operational  Availability  Component  Breakout 


NOMINAL  SRO  ANALYSIS 


DOWNTIME/ UPTIME  RATIO 


Table  11-1.  Reliability  Terminology. 

Reliability.  The  probability  that  an  item  can  perform  its 
intended  functions  for  a  specified  interyal  under  stated 
conditions. 

Mission  Reliability.  The  probability  of  completing  a 
mission  for  a  period  of  time  under  conditions  stated  in  the  ^ 
mission  profile.  Mission  reliability  addresses  failures  which 
cause  either  degradation  in  mission  performance  or  mission 
aborts . 

Logistics  Oriented  Reliability.  (Sometimes  labeled 
logistics  support  frequency)  The  rate  at  which  demands  for 
logistics  resources  are  made,  regardless  of  the  impact  on  the 
mission.  The  logistics  rates  typically  measured  are  for 
unscheduled  maintenance  actions,  essential  maintenance 
actions,  and  part  remoyals. 

Table  11-1.  Reliability  Terminology. 
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Table  11-2.  Maintainability  Terminology. 

Maintainability.  The  ability  of  an  item  to  be  retained  in 
or  restored  to  a  specified  condition  when  maintenance  is 
performed  by  personnel  haying  specified  skill  leyels,  using 
prescribed  procedures  and  resources,  at  each  prescribed  leyel 
of  maintenance  and  repair. 

Mean  Time  To  Repair  (MTTR) .  The  total  correctiye 
maintenance  action  downtimes  diyided  by  the  total  number  of 
correctiye  maintenance  actions  during  a  giyen  period  of  time. 

Maintenance  Ratio  (MR) .  A  measure  of  the  maintenance 
manpower  required  to  maintain  an  item  in  an  operational 
enyirozment .  It  is  expressed  as  the  cxmulative  man-hours  of 
corrective  and  scheduled  maintenance  expended  in  direct  labor 
during  a  given  period,  divided  by  the  cumulative  item  usage 
(e.g.,  hours,  miles,  or  cycles)  during  the  same  time. 


Table  11-2.  Maintainability  Terminology. 


Table  11-3.  Availability  Terminology. 

Availability.  A  measure  of  the  degree  to  which  an  item 
is  in  an  operable  and  committable  state  at  the  start  of  the 
mission,  when  the  mission  is  called  for  at  an  unknown  (random) 
point  in  time. 

Operational  Availability  (Ao) .  The  proportion  of  time 
a  system  is  either  operating  or  is  capable  of  operating,  when 
used  in  a  specific  manner  in  a  typical  maintenance  and  supply 
environment.  All  calendar  time  is  considered.  Ao  is 
typically  calculated  for  both  the  war  and  peace  time 
environments . 


Table  11-3.  Availability  Terminology. 
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Table  11-4.  OPERATIONAL  RELIABILITY  CLASSIFICATIONS 

Crew  Correctable  Maintenance  Action  (CCMA) .  (Optional) 

Operational  Mission  Failure  (OMF) .  Any  incident  or 
malfunction  of  the  system  that  causes  (or  could  cause)  the 
inability  to  perform  one  or  more  designated  mission  essential 
functions. 

Essential  Maintenance  Action  (EMA) .  (Optional)  Log  R 
Unscheduled  Maintenance  Action  (UMA) .  Log  R 


Table  11-4.  Operational  Reliability  Classifications 
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Chapter  12 

MANPRINT  Considerations  in  Test  &  Evaluation 


Section  I 
Introduction 


12-1.  Human  Systems  Integration  (HSI) 

DoDI  5000.2,  Defense  Acquisition  Management  Policy  and 
Procedures,  identifies  a  Human  Systems  Integration  (HSI)  program. 
This  initiative  seeks  to  reduce  dependence  upon  unrealistic  or 
unsustainable  levels  of  human  performance  and  human  resources. 

The  Army's  version  of  HSI  is  the  l^NPRINT  program  (AR  602-2). 
MANPRINT  is  an  engineering  analysis  and  management  process  to 
identify  and  articulate  requirements  and  constraints  of  human 
resources,  human  performance,  and  hazards  to  personnel  so  these 
matters  will  influence  system  design. 

12-2.  Human  Performance 

MANPRINT  in  test  and  evaluation  (T&E)  seeks  to  determine  whether 
human  performance  problems  degrade  overall  system  performance. 
Soldier  performance  on  critical  tasks  is  the  underlying  concept 
here.  In  addition  to  system  design,  procedures  can  have  profound 
effects  upon  human  performance  and  also  need  to  be  examined 
during  T&E. 

12-3.  Purpose  .  . 

This  chapter  reviews  MANPRINT  in  T&E,  identifying  what  needs  to 
be  accomplished  and  who  is  responsible  for  the  action. 
Requirements  are  presented  throughout  the  phases  of  continuous 
evaluation  (CE)  and  issue  development,  T&E  planning  and 
pifeparat ion ,  data  collection  and  authentication,  evaluation,  and 
reporting. 

12-4.  Evaluator  &  tester  relationship 

The  evaluator  is  responsible  for  MANPRINT  CE  and  evaluation  of 
test  data.  The  tester  is  responsible  for  developing  data 
collection  instruments  and  then  collecting  MANPRINT  data  during  a 
test. 

12-5.  Goal 

When  human  performance  or  other  MANPRINT  problems  are  identified, 
their  circumstances,  causes,  and  potential  solutions  need  to  be 
determined.  Solutions  can  include  changes  in  system  design, 
tactical  procedures  and  techniques,  manpower  and  organization. 
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personnel,  and  training. 


Section  II 

Continuous  Evaluation  Methodology 


12-6.  MANPRINT  issues 

MANPRINT  CE  strives  to  resolve  human  performance  issues  before  an 
Initial  Operational  Test  and  Evaluation  (lOT&E)  or  a  fielding 
decision.  The  evaluator  assesses  a  system's  MANPRINT  status  as 
part  of  the  overall  responsibility  to  perform  CE.  This  involves 
identifying  MANPRINT  issues  and  tracking  attempted  resolutions. 
Unresolved  MANPRINT  issues  ultimately  addressed  in  test  and 
evaluation  planning. 

a.  System  efficiency  and  effectiveness  issues.  CE  identifies 
problems  and  potential  changes  for  improving  total  system 
performance.  Such  changes  may  involve  modifications  in: 

(1)  Hardware  and  software  designs 

( 2 )  Manpower 

(3)  Personnel 

(4)  Training 

(5)  Procedures  and  techniques 

b.  Health/Safety  issues.  CE  also  seeks  to  identify 
detrimental  health  or  safety  effects  of  system  operation, 
maintenance,  and  support. 

12-7.  CE 

MANPRINT  CE  consists  of  two  general  activities  -  assessing  status 
and  communicating  evaluation  plans. 

a.  Assessing  MANPRINT  status 

(1)  A  major  source  of  information  for  status  assessment  is 
the  System  MANPRINT  Management  Plan  (SMMP) . 

(a)  The  SMMP  is  the  integrated  planning  document  for 
all  MANPRINT  activities  and  analyses  associated  with  system 
acquisition.  The  SMMP  includes  human  performance  issues,  data 
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sources,  and  plans  for  front-end  analyses  pEA) .  Updated  at  the 
beginning  of  each  developmental  phase,  it  influences  all 
reguirements  documents,  solicitation  documents,  and  T&E  planning 
documents.  It  also  influences  planning  documents  for  combat  and 
training  development. 

(b)  The  SMMP  must  provide  the  basis  for  developing 
testable  evaluation  issues  and  criteria  about  human  performance 
and  its  contribution  to  system  performance.  It  must  also  contain 
the  Target  Audience  Description  (TAD)  so  that  test  participant 
representativeness  can  be  determined.  (See  Figure  12—1.) 

(insert  Figure  12-1) 

(2)  The  MANPRINT  Joint  Working  Group  (MJWG)  is  responsible 
for  developing  and  updating  a  SMMP  for  every  mission  area 
deficiency  requiring  a  new  development  solution.  The  combat 
developer  [usually  the  Training  and  Doctrine  Command  (TRADOC)  or 
the  PM  for  Information  Mission  Areas  (IMA)  systems]  is 
responsible  for  forming  the  MJWG. 

(a)  The  MJWG  consists  of  representatives,  from  the 
School  and  post,  with  expertise  in  a  MANPRINT  domain.  They 
include  the  Directorate  of  Combat  Developments,  Directorate  of 
Training  and  Doctrine,  TRADOC  System  Manager  (if  appointed). 
Specialty  Proponency  Office,  Post  Safety  Office,  and  Army 
Research  Laboratory  (ARL)  —  Human  Research  and  Engineering 
Division  (HRE)  field  office. 

(b)  There  are  also  consulting  members  of  the  MJWG. 

These  may  include  representatives  from  the  Army  Materiel  Command 
(AMC) ,  the  Project  Manager's  Office,  technical  testers  and 
evaluators,  and  operational  testers  and  evaluators. 

(3)  Information  from  agencies  evaluating  specific  aspects 
of  the  MANPRINT  program  should  be  reviewed. 

(a)  The  Health  Hazard  Assessment  Report  (HHAR)  prepared 
by  the  Army  Environmental  Hygiene  Agency  (AEHA)  or  the  Medical 
Research  and  Development  Command  (MRDC) . 

(b)  The  Human  Factors  Engineering  Analysis  (HFEA) 
prepared  by  ARL-HRE.  The  HFEA  deals  primarily  with  HFE  issues, 
although  it  may  also  address  system  safety  and  training. 

(c)  The  Manpower-Personnel-Training  (MPT)  Assessment 
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prepared  by  the  Deputy  Chief  Staff  for  Personnel.  This 
assessment  is  prepared  by  the  DCSPER  for  major  systems  before 
each  milestone  review.  This  combines  the  results  of  tests, 
studies,  and  analyses  related  to  human  factors.  The  combat  and 
materiel  developers  jointly  prepare  the  MANPRINT  Assessment  for 
non-major  systems. 

(d)  The  System  Safety  Management  Plan  (SSMP)  prepared 
by  the  System*  Safety  Working  Group  (SSWG) .  SSWG  co-chairmen 
represent  the  PM  and  the  Safety  Office  of  the  affiliated  AMC 
major  subordinate  command.  SSWG  members  are  from  basically  the 
same  organizations  as  those  represented  on  the  Test  Integration 
Working  Group  (TIWG) .  The  SSMP  is  a  source  of  safety  and  health 
hazard  issues  for  consideration  in  the  System  MANPRINT  Management 
Plan  (SMMP) .  The  SSMP  guides  system  safety  engineering  and 
testing  efforts  conducted  by  the  developing  contractor. 

(e)  Safety  Assessment  Reports  (SAR)  prepared  by  the 
system  contractor  for  the  PM.  The  PM's  office,  with  the  help  of 
its  supporting  safety  office  (from  a  major  subordinate  command  of 
AMC) ,  prepares  the  System  Safety  Risk  Assessment  (SSRA) .  For 
some  systems,  the  Army  Safety  Center  provides  an  Independent 
Safety  Assessment  based  upon  the  SSRA. 

(f)  Critical  tasks  identified  by  the  combat  developer. 

(g)  The  results  of  early  operational  testing  requested 
by  combat  and  training  developers.  Such  testing  includes 
mockups,  prototypes,  or  competing  hardware  and  software. 
Applicable  tests  include  Concept  Evaluation  Program  (CEP) ,  Force 
Development  Test  and  Experimentation  (FDTE) ,  Early  User  Test  and 
Experimentation  (EUT&E) ,  Training  Effectiveness  Analysis  (TEA) , 
and  Limited  User  Tests  (LUT) .  These  tests  support: 

(1)  Identification  and  validation  of  operator  and 
maintainer  critical  tasks. 

(2)  Identification  and  development  of  testable  human 
performance  issues. 

(3)  Development  and  validation  of  Measures  of  Performance 
(MOP)  and  operational  standards  for  human  performance. 

(4)  Examination  of  Tables  of  Organization  and  Equipment 
(TOE)  design  options.  Examples  are  mixes  and  options  for 
organization,  MOS  or  Civilian  Job  Series  (CJS) ,  and  personnel 
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numbers . 

(5)  Design  and  validation  of  training  programs. 

(6)  Examination  of  operating  procedures  and  techniques. 
Procedures  can  be  in  the  form  of  tactical  Standard  Operating 
Procedures  (SOPs) . 

(7)  Examination  of  alternative  hardware  and  software 
design  features,  especially  with  prototypes  and  mockups. 

b.  Communicating  MANPRINT  aspects  of  operational  evaluation 
planning  activities  to  system  developers 

(1)  Formal  mechanisms  include  the  TEMP,  TEP,  and  periodic 
operational  assessments. 

(2)  Informal  mechanisms  include  participation  in  the 
various  working  groups  dealing  with  MANPRINT  (MJWG,  SSWG,  TIWG) . 


12-8.  MANPRINT  Evaluation  Data  Base  (MEDB) 

OEC  uses  information  from  the  previously  mentioned  sources  to 
identify  MANPRINT  issues  and  develop  OT&E  plans  beginning  early 
in  the  acquisition  process.  The  system's  MANPRINT  status  is 
then  tracked  by  maintaining  its  MEDB.  The  MEDB's  fields  are 
listed  in  Table  12-1. 

(insert  Table  12-1) 


Section  III 

DT&E  Planning  and  Preparation 


12-9.  Developmental  Test  and  Evaluation  (DT&E) 

The  DT&E  MANPRINT  goal  is  to  determine  if  the  system  design 
allows  it  to  be  operated  and  maintained  by  the  designated  number 
of  soldiers  representative  of  the  target  audience  description, 
with  the  proposed  system  training  and  under  the  expected  use 
conditions.  Developmental  tests  requiring  the  use  of  military 
personnel  to  operate,  support,  transport,  erect,  occupy,  or  use 
the  equipment  when  fielded  should  ideally  use,  as  test 
participants,  Soldier-Operator/-Maintainer  T&E  (SOMTE) 
personnel.  These  are  military  personnel  who  are  trained  as 
developmental  testers  for  assessing  the  soldier-machine 
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interface  during  DT.  The  developmental  lEP/IAP  specifies  a 
requirement  for  the  operation  and  maintenance  to  be  performed  by 
SOMTE  personnel. 

a.  Critical  Technical  Parameters  addressing  MANPRINT  issues 
are  included  in  Part  III  of  the  TEMP. 

b.  The  MANPRINT  section  of  the  developmental  lEP/IAP 
addresses  human  factors  engineering,  system  safety  and  health 
hazards,  and  the  manpower,  personnel,  and  training  elements  of 
ILS. 


c.  The  Detailed  Test  Plans  specify  detailed  test  procedures 
to  determine  if  requirements  for  human  factors  engineering,  and 
system  safety  and  health  hazards  have  been  met. 


Section  IV 

OT&E  Planning  and  Preparation 


12-10.  Operational  Test  and  Evaluation  (OT&E) 

The  major  MANPRINT  goal  for  OT  is  identification  of  system 
design  features  and  characteristics  which  adversely  affect  users 
and  maintainers.  Human  performance  issues  seldom  appear  as  a 
Critical  Operational  Issue  and  Criteria  (COIC) .  They  are 
usually  additional  operational  issues  and  criteria  (AOIC) 
supporting  effectiveness  and  suitability  COICs.  Examples  of 
these  type  AOICs  are: 

a.  Global  MANPRINT  issue  and  criterion. 

(1)  Issue:  Are  the  human  performance  and  human  factors 
characteristics  of  the  operation,  maintenance,  training,  and 
support  of  the  system  acceptable  in  training  and  operational 
settings? 

(2)  Criterion:  The  system  must  be  designed  so  that  all 
critical  tasks  related  to  its  operation,  maintenance,  and 
support  can  be  performed  by  trained  personnel  without  requiring 
either  increases  in  planned  manpower  or  training,  nor  greater 
aptitudes  and  abilities  than  expected  in  target  MOS  or  CJS 
populations.  Design  characteristics  must  not  create  short-  or 
long-term  safety  or  health  hazards. 

b.  Training  aspects  of  the  system.  Training  validity  is 
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based  on  performance  in  the  training  setting.  Performance 
validity  is  based  on  the  transfer  of  training  to  performance  in 
the  operational  setting. 

(1)  Training  Validity 

(a)  Issue;  Do  trainees  learn  the  knowledge  and 

skills? 

(b)  Criteria: 

(1)  Operator  and  maintenance  training  will  qualify 
ninety  percent  of  representative  trainees  to  perform  each 
critical  task  to  the  prescribed  training  standard. 

(2)  Collective  training  will  qualify  ninety  percent  of 
representative  crews  to  perform  each  critical  task  to  the 
prescribed  training  standard. 

(3)  Performance  Validity 

(a)  Issue:  Do  the  knowledge  and  skills  provided  by  the 
training  program(s)  enable  trainees  to  operate,  maintain,  and 
support  the  system  effectively  in  an  operationa].  setting? 

(b)  Criteria: 

(1)  Operator  and  maintainer  training  will  qualify 
ninety  percent  of  the  trainees  to  perform  each  critical  task  to 
the  prescribed  operational  performance  criterion  under  combat  or 
other  operational  conditions. 

(2)  Collective  training  will  qualify  ninety  percent  of 
trained  crews  to  perform  each  critical  task  to  the  .prescribed 
operational  performance  criterion  under  combat  or  other 
operational  conditions. 

c.  Human  Factors  Engineering  (HFE)  aspects  of  the  system. 

(1)  Issue:  What  system  design  features  adversely  affect 
the  ability  of  representative  participants  to  operate,  maintain, 
and  support  the  system  in  an  operational  setting? 

(2)  Criterion:  None,  this  is  an  investigative  issue. 

(Not  enough  is  known  to  completely  specify  all  relevant  MOPs  and 
their  standards  as  individual  criteria.) 
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d.  Manpower  and  organizational  aspects  of  the  system. 

(1)  Issue:  Does  the  applicable  TOE  provide  manpower 
resources  for  the  effective  operation,  maintenance,  and  support 
of  the  system? 

(2)  Criterion:  None,  this  is  an  investigative  issue. 

e.  Personnel  and  personnel  selection  aspects  of  the  system. 

(1)  Issue:  Can  the  system  be  effectively  operated, 
maintained,  and  supported  by  personnel  in  the  MOS(s)  or  CJS(s) 
planned  for  the  system? 

(2)  Criterion:  None,  this  is  an  investigative  issue. 

f.  Operational  procedures  and  techniques  aspects  of  the 
system. 

(1)  Issue:  Do  unit  tactical  SOPs  provide  operational 
procedures  and  techniques  in  enough  detail  to  permit  trained 
individuals  and  crews  to  operate,  maintain,  and  support  the 
system  effectively  in  an  operational  setting? 

(2)  Criterion:  This  is  an  investigative  issue. 

g.  System  safety  and  health  hazards. 

(1)  Issue:  Is  the  system  safe  for  personnel  to  operate, 
maintain,  and  support  in  operational  and  training  settings? 

(2)  Criterion:  There  must  be  no  safety  or  health  hazards 
associated  with  the  training,  operation,  maintenance,  or  support 
of  the  system  having  a  Risk  Assessment  Code  (RAC)  with  higher 
risk  than  IV-A,  III-B,  III-C,  II-D,  and  I-E. 

Note:  The  RAC  concept  is  described  in  MIL-STD-882B.  The 
particular  RACs  used  in  a  criterion  should  conform  to  the  risk 
management  matrix  prescribed  in  the  SSMP  and  to  the  acquisition 
strategy  (also  see  AR  40-10) . 

12-11.  The  operational  evaluator 

The  operational  evaluator  takes  the  following  steps: 

a.  Identifies  MANPRINT  issues  not  resolved  prior  to  test. 
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b.  Develops  criteria  and  MOPs  for  these  issues-. 

c.  Ensures  MANPRINT  evaluation  costs  are  contained  in  the 
Outline  Test  Plan  (OTP) . 

d.  Completes  the  MANPRINT  portion  of  chapter  two  of  the  TEP. 

e.  Reviews  chapter  three  of  the  TEP  to  ensure  proposed  data 
elements  and  their  collection  instruments  and  procedures  meet 
MANPRINT  evaluation  requirements. 

f.  Makes  a  military  subject  matter  expert's  judgement  of 
whether  Safety  Release  (SR)  restrictions  permit  adequate 
training  and  operational  testing. 

g.  Evaluates  individual  and  unit  performance  during  pretest 
training  in  a  manner  consistent  with  OT&E  scenarios  and 
performance  measures. 

12-12.  The  operational  tester 

The  operational  evaluator  takes  the  following  steps? 

a.  Identifies  MANPRINT  data  requirements,  instruments,  and 
collection  procedures.  Sample  data  items  in  Figure  12-2  are  a 
topic  menu  for  developing  MANPRINT  data  and  instrumentation 
requirements  during  test  planning. 

(insert  Figure  12-2) 

b.  Ensures  resources  to  support  MANPRINT  data  collection  and 
analysis  (level  three)  are  contained  in  the  OTP. 

c.  Reviews  chapter  two  of  the  TEP  and  responds  in  the 
MANPRINT  portion  of  chapter  three. 

d.  Ensures  that  the  training  developer  knows  the  scope  of 
operational  test  activities.' 

e.  Ensures  that  the  trainer  issues  the  Operational  Test 
Readiness  Statement  (OTRS)  before  the  pilot  test  begins. 

f.  Ensures  that  test  participants  are  representative  of  the 
MOS  or  CJS  populations  for  operators,  maintainers,  and 
supporters.  Given  test  player  social  security  numbers,  the 
Defense  Manpower  Data  Center  (DMDC)  will  compare  them  with  all 
others  in  their  MOS  or  CJS. 
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g.  Ensures  that  the  test  design  has  been  approved  by  a  Human 
Use  Committee  (HUC)  (AR  70-25)  before  the  pilot  test  begins. 


Section  V 

System  Safety  and  Health  Hazards 


12-13.  Requirements  for  Release 

Systems  will  not  be  released  to  field  troops  or  for  testing  by 
troops  until  the  following  requirements  are  accomplished  (AR 
385-16  and  AR  40-10) . 


a.  Evaluations  of  the  system  safety  and  health  hazards  of 
each  item  or  system  will  be  conducted  throughout  the  T&E 
program.  Safety  testing  will  be  conducted  in  accordance  with 
approved  international  test  operating  procedures  or  TECOM  test 
operating  procedures  for  the  specific  system  under  test.  The 
testing  will  assess  personnel  and  equipment  hazards  inherent  in 
the  system  and  its  associated  operation  and  maintenance  hazards. 
This  testing  will  be  considered  early  in  the  test  -  planning 
cycle  and  will  continue  throughout  the  acquisition  process  as  an 
element  of  the  normal  test  program  for  evaluation  at  each 
milestone.  Special  attention  will  be  given  to  verifying  the 
adequacy  of  safety  and  warning  devices  and  other  measures 
employed  to  control  hazards.  The  adequacy  of  hazard  warning 
labels  on  equipment,  and  warnings,  precautions,  and  control 
procedures  in  equipment  publications,  will  be  evaluated. 
Moreover,  commanders  of  the  user  units  must  be  made  aware  of  and 
agree  to  the  safety  and  health  hazards  to  which  their  units  will 
be  exposed. 

b.  Pertinent  data  from  all  testing  (including  contractor 
testing)  will  provide  a  basis  to  evaluate  safety  and  health 
characteristics.  In  addition,  separate/specific  safety  and 
health  tests  will  be  performed  on  hazardous  devices,  components, 
or  by-products  to  determine  the  nature  and  extent  of  hazards 
presented  by  the  materiel.  Particular  attention  will  be  given 
to  identifying  and  evaluating  special  safety  and  health  hazards 
presented  by  radioactive  materials,  radio  frequency  emitters, 
toxic  gases,  laser  devices,  toxic  and  carcinogenic  materials, 
gaseous  emissions,  blast  overpressure,  and  harmful  noise 
sources.  Health  issues  that  cannot  be  resolved  within  the  T&E 
community  will  be  elevated  to  the  Surgeon  General  in  accordance 
with  AR  40-10. 
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c.  Evaluators  will  ensure  that  T&E  documentation  provides  an 
independent  assessment  of  hazards  and  that  the  results  of  the 
safety  evaluations  are  included  in  all  evaluations.  The  ^ 
developmental  lER  will  contain  a  safety  confirmation  as  required 
by  AR  385-16. 

d.  Test  organizations  will  ensure  that: 

(1)  The  DTPS  for  all  developmental  tests  provide  a 
developmental  assessment  of  hazards,  and  the  results  are 
included  in  all  TRs. 

(2)  Precautions  are  taken  to  protect  personnel  and 
equipment  during  tests. 

(3)  A  safety  assessment  report  is  used  to  integrate 
safety  into  test  planning  and  procedures  and  for  shipping  and 
handling  the  system. 

(4)  Government  system-level  testing  does  not  begin  until 
a  safety  assessment  report,  prepared  by  the  program  manager,  has 
been  received,  reviewed,  and  accepted  by  the  test  organization. 

(5)  Pretest  training  for  operational  testing,  and  any 
other  testing  that  uses  soldiers  not  assigned  to  the  test 
organization  as  test  players,  will  not  begin  until  a  safety  ^ 
release  has  been  reviewed  and  accepted  by  the  test  organization. 

(6)  Test  plans  are  reviewed  by  the  appropriate  Human  Use 
Committee  (HUC) . 


Section  VI 
DT&E  Execution 

12-14.  DT&E  data  collection 

During  developmental  testing,  the  following  MANPRINT-r elated 
data  is  collected: 

a.  Objective  measurements  (i.e.,  sound,  lighting,  vibration, 
workspace,  temperature/humidity/ventilation)  as  required  to 
document  the  adequacy  of  the  soldier-machine  interface. 

b.  Human  performance  measurements  for  SOMTE  personnel 
completing  operational  and  maintenance  tasks  relating  to  mission 
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performance. 

c.  Questionnaire  and  interview  administration  to  obtain 
SOMTE  personnel  subjective  input  on  ease  of  operation  and 
maintenance,  and  adequacy  of  design  to  complete  mission 
requirements. 

d.  Questionnaires  and  interviews  to  determine  the  SOMTE 
personnel's  subjective  input  on  the  adequacy  of  new  equipment 
training  (N£t) . 


Section  VII 
OT&E  Execution 


12-15.  OT&E  data  collection 

The  tester  collects  data  on  human  performance  and  other  MANPRINT 
concerns  during  OT. 

a.  Data  collection  efforts  include: 

(1)  Using  health  and  safety  related  procedures  as 
required  by  issues  and  criteria,  the  SR,  and  the  HUC. 

(2)  Making  direct,  accurate,  objective,  and 
non-interfering  measurements  of  player  performance.  These  are 
time  and  accuracy  measures  of  behavior  related  to  MOPs. 
Measurements  include  both  individual  and  group  performance. 

(3)  Video  taping  soldier  activities,  particularly  where 
they  can  not  be  directly  observed. 

(4)  Getting  observations  and  ratings  from  expert 
observers,  including  observations  about  videotaped  activities. 

(5)  Integrating  MANPRINT  data  into  the  system's  overall 
test  data  base: 

(a)  Performance  data 

(b)  Users'  opinions  about  the  operating  characteristics 
of  hardware  and  software  design  features  and  operating 
procedures 

(c)  Individual  characteristics  of  test  participants 
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(aptitude  test  scores,  anthropometric  measures,  etc.) 
b.  Provides  level  three  data  to  OEC. 

(1)  Automated  data  base  with  linlced  files  containing 
related  data  elements. 

(2)  Reduced  manual  data  with  indexed  printouts, 
spreadsheets,  etc. 

(3)  Edited  audio/ video  tape. 

12-16.  Data  Authentication  Group  (DAG)  MANPRINT  Procedures 


a.  DAGs  are  sometimes  formed  to  help  ensure  the  validity  of 
operational  test  data.  This  DAG  process  should  include  all 
problems  affecting  players'  ability  to  use,  repair,  or  support 
the  system.  Such  problems  may  involve  individual  or  unit 
performance,  human  resources,  or  hazards  to  personnel  and 
equipment.  DAG  members  determine  the  nature  of  MANPRINT 
problems  and  their  anticipated  mission  impacts. 

(1)  The  problem  statement  describes  difficult  or  below- 
standard  human  or  system  performance  as  described  in  the 
MANPRINT  Problem  Report  (Figure  12-3). 

(insert  Figure  12-3) 

(2)  Potential  impact  upon  mission  accomplishment  is  based 
on  the  MANPRINT  Problem  Impact  Rating  Scale  presented  in  Table 
12-2. 

(insert  Table  12-2) 

b.  The  following  procedures  for  the  tester  help  insure  that 
human  performance  problems  are  adequately  examined  before  the 
Test  and  Evaluation  Report  (TER)  identifies  them  as  such. 

(1)  Complete  the  MANPRINT  Problem  Report  as  soon  as 
possible  after  a  human  performance  problem  is  detected. 

(2)  Tester's  MANPRINT  analyst  debriefs  test  participants, 
data  collectors,  and  other  relevant  test  directorate  personnel. 
The  analyst  discusses  each  problem  report  to  confirm  the 
validity  of  its  contents.  The  analyst  then  compiles  individual 
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MANPRINT  Problem  Reports  into  an  overall  draft  DAG  MANPRINT 
Report.  This  summary  uses  the  format  in  Figure  12-4. 

(insert  Figure  12-4) 

(3)  Each  DAG  member  independently  rates  problem  severity 
(Table  12-2) .  Ratings  are  then  summarized  for  all  DAG  members 
and  each  problem  is  assigned  the  most  commonly  given  rating. 
Problems  are  then  rank  ordered  from  most  through  least  severe. 

(4)  This  preliminary  determination  is  presented  to  DAG 
members  to  identify  any  substantial  objections  to  the  majority's 
rating.  The  rationale  for  such  objections  (if  any)  is  heard. 

The  DAG  members  may,  as  a  group,  choose  to  change  the  majority's 
rating.  The  final  version  becomes  the  DAG  MANPRINT  Report. 


Section  VIII 

Analyzing,  Evaluating,  and  Reporting  DT&E  Findings 


12-17.  MANPRINT  analyses 

The  developmental  independent  evaluator/assessor  report  their 
analyzes  of  MANPRINT  issues  in  the  lER/IARs  as  follows: 

a.  The  human  factors  engineering,  system  safety  and  health 
hazard,  and  the  manpower,  personnel,  and  training  elements  of 
ILS  test  data  from  all  sources  are  analyzed  to  determine 
conformance  to  appropriate  requirements  documents. 

b.  Any  failures  to  meet  the  requirements  are  then 
assessed/evaluated  to  determine  their  cause  and  their  effect  on 
overall  mission  performance. 

c.  The  test  data  and  analyses  elements  of  the  lER/IAR 
regarding  human  factors  engineering,  system  safety  and  health 
hazard,  and  the  manpower,  personnel,  and  training  elements  of 
ILS  are  reviewed  for  MANPRINT-related  findings  and  the  impact  of 
the  problems  are  summarized  in  a  separate  MANPRINT  Summary 
Statement  in  the  lER/IAR. 


Section  IX 

Analyzing,  Evaluating,  and  Reporting  OT&E  Findings 
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12-18.  Operational  evaluator  findings 

The  operational  evaluator  does  the  following: 

a.  Analyze  player  performance  impact  on  overall  system  and 
unit  performance. 

(1)  Analyze  player  performance  measures  for  impact  by 
procedures,  equipment  design  features,  and  system  operating 
characteristics . 

(2)  Analyze  player  performance  measures  for  sensitivity 
to  individual  differences.  Examples  are  aptitude,  education, 
training,  and  physical  characteristics. 

(3)  Conduct  additional  analyses  of  data  required  to 
address  all  MANPRINT  MOPs. 

b.  Evaluate  results  in  terms  of  MOP  findings  contributing  to 
criteria  which  in  turn  address  evaluation  issues.  Issues 
impacted  by  MANPRINT  typically  involve  system  efficiency  and 
effectiveness  along  with  health/safety.  The  evaluation 
describes  and  assesses  human  performance  problems  which  degrade 
overall  system  performance. 

(1)  HFE.  System  design  features  failing  to  accommodate 
human  characteristics  and  limitations. 

(2)  Manpower /Organizational. 

(a)  Personnel  demands  exceeding  resources  in  applicable 

TOE. 

(b)  MOS  or  CJS,  skill  levels,  and  grades  inadequate  to 
sustain  operations. 

(c)  Task  organization,  procedures,  and  techniques 
inappropriate  for  performing  required  operation,  maintenance, 
and  support  tasks. 

(3)  Personnel.  MOSs  or  CJSs  for  operating,  maintaining, 
and  supporting  the  system  that  do  not  possess  appropriate  and 
adequate  aptitudes. 

(4)  Training.  Critical  tasks  untrained  or  inadequately 
trained. 
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(5)  System  safety  and  health  hazards. 

(a)  Inadequate  hazard  fixes. 

(b)  Residual  safety  and  health  hazards. 

c.  Findings  are  then  evaluated  to  identify  potential 
solutions.  These  can  involve  modifications  to  any  or  all  of  the 
following  aspects  of  human  performance. 

(1)  Hardware  and  software  designs. 

(2)  Manpower. 

(3)  Personnel. 

(4)  Training. 

(5)  Procedures  and  techniques. 

d.  Write  and  publish  the  MANPRINT  portion  of  the  TER  or 
Operational  Assessment  (OA) . 
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SMMP  REVIEW  GUIDE 

a.  References 

(1)  AR  602-2  Manpower  and  Personnel  Integration 
(MANPRINT)  in  the  Materiel  Acquisition  Process,  19  Apr  1990, 
as  modified  by  DA  Memorandum  "MANPRINT  Procedural  Change; 
System  MANPRINT  Management  Plan  (SMMP)  Format  Revision," 

Harold  Booher,  27  Sep  1991. 

(2)  DoD  Memorandum  "Human  Systems  Integration  Plan 
Implementation  Procedures,"  Assistant  Secretary  of  Defense, 
Christopher  Jehn,  28  May  91. 

b.  SMMP  Element  1,  TITLE  PAGE 

(1)  Was  the  SMMP  approved  60  days  prior  to  each 
Milestone  Decision  Review? 

C.  SMMP  Element  2,  ABBREVIATED  TOTAL  SYSTEM  DESCRIPTION 

(1)  Is  the  operational  environment  adequately 

described? 

(2)  Does  the  TAD  contain  all  relevant  MOSs, 
including  operators,  maintainers,  and  support  personnel? 

(3)  Does  the  TAD  contain  sufficient  information  to 
assess  whether  test  soldiers  are  representative  of  target  MOS 
populations? 

d.  SMMP  Element  3,  ACQUISITION  STRATEGY 

(1)  Is  the  program  category  and  acquisition  strategy 
(developmental,  nondevelopmental,  materiel  change)  indicated? 

e.  SMMP  Element  4,  PREDECESSOR  DEFICIENCIES 

(1)  Have  predecessors  for  each  component  of  the 
system  been  considered,  including  training  devices  and  repair 
and  support  equipment? 


Figure  12-1.  SMMP  Review  Guide 


(2)  If  there  is  no  direct  predecessor  system,  have 
comparable  components  been  considered  to  construct  a  notional 
system  for  planning  purposes? 

f.  SMMP  Element  5,  MANPRINT  PARAMETERS 

(1)  Are  there  any  constraints  on  MOSs  or  existing 
force  structure? 

(2)  Has  a  crew  size  been  previously  decided? 

(3)  Will  the  system  be  contractor-supported  for  the 
life  of  the  system? 

(4)  Will  there  be  ’’package  fielding”? 

g.  Element  6,  MANPRINT  ISSUES 

(1)  Are  system  and  human  performance  issues  defined 
so  that  they  can  be  operationally  tested? 

(2)  Are  system  and  human  performance  issues  defined 
so  the  system  and  its  developers  (rather  than  soldiers  manning 
the  system)  are  held  responsible  for  the  achievement  of 
required  levels  of  system  performance? 

(3)  Are  task  analyses  planned?  Will  findings  be 
available  in  time  to  affect  system  design? 

(4)  Will  Training  Effectiveness  Analyses  (TEA)  be 
available  to  validate  training  programs  before  lOTE? 

h.  Element  7,  MANPRINT  EXECUTION 

(1)  Are  task  analyses  planned? 

(2)  Are  there  planned  TRADOC  test  activities  (CEP, 
FDTE,  TEA,  and  EUTE)  to  answer  questions  about  options  for 
organizations  and  training  programs? 


Figure  12-1  SMMP  Review  Guide  (cont'd) . 
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Element  8,  Annex  A,  COORDINATION 


(1)  Have  combat,  training,  and  materiel  developers 
been  identified? 

(2)  Has  the  Logistics  Evaluator  been  identified? 

(3)  Is  the  Human  Engineering  Laboratory  (HEL) 

included? 

(4)  Is  the  Surgeon  General  included? 

(5)  Is  the  Army  Research  Laboratory  included? 

j.  The  following  DoD  requirements  also  apply  to  SMMPs. 

(1)  By  Concept  Demonstration  Approval,  does  the 
SMMP  address  the  following  MANPRINT  aspects  of  the  system? 

(a)  high  drivers  and  lessons  learned  from 
predecessor  or  comparable  system (s) 

(b)  parameters  documented  in  the  ORD  included 
in  the  Acquisition  Program  Baseline 

(c)  exit  criteria 

(d)  TAD  for  operators  and  maintainers 

(e)  impacts  accompanying  design  alternatives 

(f)  cost,  schedule,  and  design  risk 
identification  and  management 

(g)  inclusion  in  early  OT&E  during 
Demonstration  and  Validation  (D&V) 

(h)  tools,  analyses,  data  bases,  and 
methodologies  during  D&V 

(i)  incorporation  in  acquisition  strategy 

Figure  12-1  SMMP  Review  Guide  (cont'd) . 
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(2)  By  Development  Approval,  does  the  SMMP  address 
the  following  MANPRINT  aspects  of  the  system? 

(a) 

trade  offs  made  during  D&V 

concept 

(b) 

early  test  results  influence  on  design 

(c) 

risk  management  plan 

(d) 

resource  adequacy  to  support  the  program 

(e) 

criteria  to  be  included  in  OT&E 

suitability 

(f) 

source  selection  criteria  impact  on 

’  (3)  By  Production  Approval,  does  the  SMMP  address 

the  following  MANPRINT  aspects  of  the  system? 

(a) 

findings  from  OT&E 

demonstrations 

(b) 

scheduling  of  realistic  maintainability 

(4)  By  Major  Modification  Approval,  does  the  SMMP 
address  the  following  MANPRINT  aspects  of  the  system? 

ownership 

(a) 

opportunities  to  reduce  the  cost  of 

safety  hazards 

(b) 

efforts  to  correct  residual  health  and 

(c)  plans  to  transition  from  contractor  to 
organic  support  (if  applicable) 

Figure  12-1  SMMP  Review  Guide  (cont'd) . 
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SAMPLE  MANPRINT  DATA  ITEMS 
a.  Human  factors  engineering  considerations 

(1)  The  nature  of  required  soldier  performance  on 
critical  tasks  [discrete  tasks,  continuous  tasks  - 
tracking/ steering,  complex  tasks  -  vigilance/reaction, 
response  rates,  response  modes  (hands,  eyes,  feet,  voice, 
head,  torso,  etc.)f  reaction  times,  completion  times, 
accuracy,  error  rates  and  types  (omission,  commission) , 
interference  by  elements  of  task  environment] 

(2)  Equipment /Hardware  characteristics 

(a)  Efficiency  of  Soldier-Machine  Interface 
[information  displays  (visual  -  analog/ digital,  auditory, 
tactile,  proprioceptive,  olfactory),  symbology  (standard  vs. 
unique) ,  operator  controls,  speech  intelligibility  and 
recognition] 

(b)  Maintainability  [comfort,  convenience, 
portability  (bulk,  weight,  load  distribution,  straps, 
handles) ,  accessibility  for  operation  and  maintenance, 
compatibility  with  soldier  characteristics  and  states, 
physical  workload  demands] 

(3)  Mental  workload/ information  processing  demands 

(4)  Physical  workload  demands/ task  overload 

(5)  Software  interface  characteristics  [video  hardware 
(flicker,  glare,  contrast  ratio,  luminance,  resolution), 
screen  layout  and  contents  (screen  size,  partitions,  labels, 
messages,  error  statements,  data  entry /display) ,  keyboard 
characteristics  (keystroke  feedback,  key  labeling,  keyboard 
slope,  key  rollover) ,  system  feedback  (display  update  rate, 
response  time,  job  aids,  status  information),  ergonomics 
(working  level,  viewing  distance,  reach,  head  position, 
printer] 


Figure  12-2.  Sample  MANPRINT  Data  Items 


(6)  User  -  system  interaction  [interference  with  other 
systems,  training  difficulty,  error  probability  and  recovery 
ease,  message/data  prioritizing,  decision  and  information 
processing  workload] 

(7)  Task  characteristics  [task  procedures /rules, 
doctrine,  tactics] 

(8)  Task-interdependence  of  crew  members  [independent, 
parallel,  sequential,  reciprocal/ interactive,  communication 
patterns /informat ion  flow,  task-interdependence  of  crews  with 
other  organizational  elements] 

(9)  Allocation  of  system  functions  (soldiers,  hardware, 
software) 

(10)  Task  environment  and  workspace 
[availability /adequacy,  dimensions,  ingress/egress,  mobility, 
individual  clothing,  equipment,  and  weapons,  NBC  and  cold 
weather  protective  gear,  weather  conditions  (heat,  cold,  ram, 
humidity,  ice),  ambient  light  and  illumination,  air,  gases  and 
fumes,  pollutants,  vibration  -  frequency,  duration, 
acceleration  -  magnitude,  direction,  sign,  etc.,  noise 
(magnitude,  frequency,  duration,  type  {impulse  vs. 
continuous},  physical  workspace  (individual  and  crew),  stowage 
(mission  gear,  personal  equipment,  emergency  equipment) , 
accessibility  (operation  and  maintenance) ,  temperature 
(monitor  and  controls) ] 

(11)  Effects  of  system  operation  upon  soldiers 
[physical  (skills,  fatigue),  psychological  (motivation, 
morale,  satisfaction/frustration,  troop  acceptance  of 
hardware,  software,  and  task  procedures) ,  physiological 
(hearing,  motion  sickness),  cognitive  (learning,  knowledge)] 

b.  Manpower / force  structure/organization  considerations 
[numbers  of  soldiers  required  to  accomplish  tasks  and  missions 
(gross  workload) ,  TOE,  MOS,  skill  level  (over— qualified? 
under-qualified?) ,  grade,  skill  identifiers,  QQPRI/MOS 
decision,  BOIPJ _  _ 

Figure  12-2.  Sample  MANPRINT  Data  Items  (cont'd) 


c.  Personnel  considerations 


(1)  Availability  projections  (TAPA  Force  Management 
Book)  [MOS  structures  (Career  Management  Field) ,  demographic 
data,  individual  soldier  characteristics] 

(2)  Target  Audience  Description  (characteristics  of 
target  MOS  populations) 

(3)  Test  soldier  representativeness/typicality 

(4)  Aptitudes  (ASVAB) 

(5)  Test  score  categories  (AFQT) 

(6)  Training,  skill,  experience 

(7)  Anthropometries  (physical  dimensions) 

(8)  Physical  condition/abilities  (MEPSCAT) 

(9)  Medical  profile  (PULHES) 
d.  Training  considerations 

(1)  Positive/negative  transfer? 

(2)  Training  conditions  sufficiently  similar  to  job 
conditions? 

(3)  Skill  levels  adequate  for  job? 

(4)  Nonessential  skills  trained? 

(5)  Essential  skills  not  trained? 

(6)  Critical  task  lists  and  inventories 

(7)  Opportunities  to  practice 

_ (8)  Types  of  training _ 
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(9)  Individual/collective 

(10)  Institutional/unit 

(11)  Peacetlme/inobllization 

(12)  Skill  maintenance  and  retention/retraining 

(13)  Transition  training 

(14)  Training  courses. (length,  proponency) 

(15)  Training  hardware/devices/aids 

(16)  Embedded  training 

(17)  Feedback  processes  and  mechanisms 

(18)  MOS  certification 

(19)  POI 

(20)  Technical  documentation 

(21)  ICTP 

(22)  NET/NETP 

(23)  TMs 

(24)  soldier's  manuals 

(25)  SQT 

(26)  ARTEP 

e.  Safety  and  health  hazard  considerations 

(1)  Air  quality  (humidity,  temperature,  pressure,  wind 
speed,  ventilation) _ 

Figure  12-2.  Sample  MANPRINT  Data  Items  (cont'd) 
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(2)  Pollutants  (density  of  gases,  particles,  fumes, 
chemicals,  organisms) 

(3)  Vibration  (magnitude,  frequency,  combined  results 
of  multiple  sources  of  vibration) 

(4)  Acceleration/deceleration  (frequency,  duration, 
interval) 

(5)  Noise  (pulse,  steady  state,  frequency,  intensity, 
exposure  rate,/ duration) 

(6)  Light/Illumination/Lasers  (glare,  luminance, 
contrast,  frequency,  power) 

(7)  Blast 

(8)  Electromagnetic  Radiation  (EMR)  hazards 

(9)  Lightning 

(10)  Electrostatic  Discharge  (ESD) /Electromagnetic 
Interference  (EMI) 

(11)  Electromagnetic  Compatibility  (EMC) 

(12)  Electrical/mechanical  hazards 

(13)  Warhead  and  explosive  hazards 

(14)  Fuels  and  flammable  materials 

(15)  Gas  toxicity 

(16)  Rough/safe  handling 

(17)  Life  support /environmental  hazards 

(18)  Adverse  weather  conditions  (heat,  cold,  rain, 

sleet,  ice,  snow,  wind,  water,  etc.) _ 


Figure  12-2.  Sample  MANPRINT  Data  Items  (cont'd) 


(19)  NBC  conditions  (individual  protective  measures, 
hardware  component/ subsystem  protective  measures, 
decontamination  procedures,  soldier  survivability) 

(20)  Safety  release  for  training  and  testing 

(21)  Fatigue 

Figure  12-2.  Sample  MANPRINT  Data  Items  (cont'd) 
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1 .  System  Name : 


(Twenty  digits  maximMP/  name  of 
system  under  test.) 


2.  Problem  No. _  (Four  digits  maximum,  a  number 

identifying  the  problem  and  assigned 
in  the  order  that  the  problem 
occurred  during  the  test.  First 
problem  is  No.  0001) 


3.  Variant: _ ^Variant  No.  1 

_ ^Variant  No.  2 

Variant  No.  nn 


(If  there  is  more  than  one  variant  to 
a  system  each  variant  is  assigned  to 
a  2-digit  no.  and  listed  here.  A 
check  mark  is  placed  next  to  the 
variant  to  which  this  Problem  Report 
refers. ) 


4.  Problem  Title; _  (40  digits  maximum,  short  name 

identifying  problem.) 

5.  Problem  Definition: 

(A  detailed  description  of  the  problem,  giving  all  salient 
points  and  where  possible  referring  to  attached  illustrations, 
photographs  and  diagrams.) 


6.  Contributing  Causes: 

a.  (A  list  of  ALL  probable  causes  of  the  problem.) 

b. 


MANPRINT  PROBLEM  REPORT 
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Pages 


7.  Probable  Consequences; 


(A  list  of  the  PROBABLE  implications  that  the  problem  has 
for  mission  accomplishment,  in  terms  of  either  direct 
effects  upon  system  performance  or  indirect  effects  upon 
the  system  performance  due  to  soldier  and  death,  injury, 
illness,  fatigue,  frustration,  or  boredom.) 


8.  Potential  List  of  Solutions: 

a.  (List  as  many  potential  solutions  as  possible,  in  order  to 
provide  the  materiel,  combat,  and  training  developers  with 
all  feasable  options  to  alleviate  the  problem.  Try  to 
identify  solutions  that  maximize  system  performance 
improvements  while  minimizing  the  performance  burden  which 
is  imposed  upon  soldiers.) 


9.  Data  Source (s): 


(List  ALL  data  sources  IN  DETAIL,  e.g.,  list  interview 
instrument  and  item  number,  OTIR  number,  of  where  details 
of  observations  can  be  found  in  the  data  base.  It  is 
absolutely  essential  to  have  a  clear  audit  trail  to  the 
raw  data . ) 


MANPRINT  REPORT 


TABLE  12-1.  MANPRINT  EVALUATION  DATA  BASE 


MANPRINT  EVALUATION  DATA  BASE 

SYSTEM  ABBREVIATION:  FILE  UPDATED: 

SYSTEM  NAME: 

PREDECESSOR  SYSTEM (S) : 

EVALUATION  DIR: 

EVALUATOR: 

OEC  AO:  TEXCOM  AO: 

EVALUATE:  ABBREVIATED  EVALUATE:  NEXT  M  (0-IV) 

NEXT  OT:  DATES:  /  /  /  / 

STATUS 

(R-A-G)_£ _ ISSUE _ 


TABLE  12-1.  MANPRINT  EVALUATION  DATA  BASE 

(cont'd) 


MANPRINT  INDIVIDUAL  ISSUES 

SYSTEM: 

ISSUES 

MISSION  IMPACT  (A-E) :  SAFETY  HAZARD  LEVEL 

ISSUE/PROBLEM  DESCRIPTION: 

SYSTEM  COMPONENT: 

CONTRIBUTING  CAUSE ( S) : 

DESCRIPTION  OF  MISSION  IMPACT: 

POTENTIAL  SOLUTION: 

DATA  SOURCES  &  THEIR  DATES: 

ACTION (S)  REQUIRED: 

TIMELINE (S) : 

RESPONSIBLE  AGENCY: 

OTP: 

TEMP/COIC  SECTION(S): 

HUC  REQUIREMENT ( S ) : 

(S)  : 


ISSUE  #: 
RAG: 

(I-IV;  A-E) 


CRITERION (lA) : 


TABLE  12-2.  PROBLEM  IMPACT  RATING  SCALE 


PROBLEM  IMPACT  RATING  DATA 


DEGREE  OF  IMPACT  OF  THE 
MANPRINT  PROBLEM  ON 
OPERATIONAL  MISSION 
ACCOMPLISHMENT? _ 

A.  SEVERE  The  problem  has  a  severe  impact  on 

mission  and  system  performance 
leading  to  a  high  probability  of 
mission  failure,  severe  damage  or 
loss  of  equipment,  or  severe  injury 
or  death  to  personnel. 


B.  MAJOR 


C.  MODERATE 


D.  MINIMAL 


E.  NEGLIGIBLE 


The  problem  has  a  major  impact  on 
mission  and  system  performance, 
leading  to  a  high  probability  of 
degraded  mission  performance,  major 
damage  to  equipment,  or  serious 
injury  to  personnel. 

The  problem  has  a  moderate  impact  on 
system  performance,  leading  to  a 
high  probability  of  degraded  mission 
performance.  There  may  be  no 
measurable  impact  upon  system 
performance,  although  there  is  a 
measurable  impact  upon  the 
performance  of  system  components  or 
subsystems  (including  .the  human 
subsystem  as  soldiers  try  to 
compensate  for,  or  work  around 
system  defects. 

The  problem  has  a  minimal  impact  on 
system  performance.  There  is  no 
measurable  impact  on  the  performance 
of  system  components  or  subsystems 
(including  the  human  subsystem) 
although  soldiers'  negative 
attitudes  toward  features  of  the 
system  may  be  measurable. 

The  problem  has  a  negligible  impact 
on  short-term  system  performance. 
There  may  be  no  measurable  impact  on 
soldier's  attitudes. 


I  a**  32. 
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Chapter  13  ,  ^ . 

Threat  Considerations  in  Test  and  Evaluation 


Section  I 
General 


13-1.  Introduction  .  . 

This  chapter  explains  the  processes  for  developing  the  threat  , 
integrating  it  into  the  T&E  planning  process,  and  portraying  it 
in  the  test.  AR  73-1,  Test  and  Evaluation  Policy,  requires  that 
testing  must  include  an  accurate  representation  of  the  threat 
projected  to  exist  at  a  system  post-initial  operational 
capability  (IOC)  date. 

13-2.  Threat  Provisions  in  Test 

Threats  must  be  identified,  approved,  and  updated  continuously 
throughout  the  life  cycle  (AR  381-11).  DA-approved  (DA  DCSINT) 
threat  or  system-specific  threat  definitions  developed  in 
accordance  with  appropriate  regulations  will  be  employed  when 
tests  are  planned,  designed, and  conducted. 

13-3.  Threat  in  the  TEMP 

V  a.  Representations  of  threats  used  for  T&E  will  be  approved 

in  accordance  with  AR  381-11,  table  2-1,  and  identified  in  the 
TEMP.  Approval  for  their  use  will  be  part  of  the  TEMP 
coordination  and  approval  process.  The  TEMP  is  a  key  program 
document  which  has  the  primary  purpose  of  describing  the 
necessary  DT  and  OT  that  relates  program  schedule,  test 
management  structure,  and  required  resources  to  COIC  or  exit 
criteria,  critical  technical  character¬ 
istics,  required  operational  characteristics,  and  evaluation  and 

decision  milestones. 

b.  One  aspect  of  the  TEMP  relates  test  events  to  threat 
intelligence,  as  depicted  in  the  STAR/STA,  in  order  to  identify 
requirements  for  all  categories  of  threat  simulators/ 
targets  ano  simulations.  DoD  Manual  5000. 2M  requires  that 
threat  system  and  simulator  requirements  be  identified  by  type, 
number,  and  availability  in  Part  V  (T&E  Resource  Summary)  of  the 
TEMP.  Also  required  is  a  comparison  with  available  projected 
threat  systems  or  simulators  and  a  statement  which  identifies 
major  shortfalls.  Target  requirements  are  to  be  treated  in  a 
similar  manner.  This  data  is  available  to  the  TIWG/PM  through 
the  report  prepared  by  the  TAWG  during  the  accreditation  of  the 
threat  simulators/ targets  for  integration  in  the  TEMP. 

/" 
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13-4 .  Threat  Considerations 

Specific  threat  considerations  that  must  be  addressed  are 
described  as  follows: 

a.  Testing  must  accurately  represent  the  threat  projected  to 
exist  at  post-IOC.  The  post-initial  operational  capability  year 
established  by  the  operational  tester  will  be  used  as  the  basis 
to  determine  threat-projection  requirements. 

b.  The  threat  integrator  member  of  the  TIWG  will  review 
threat  support  to  testing  as  part  of  the  Threat  Coordinating 
Group  process. 

c.  If  the  threat  (as  described  in  the  STAR),  or  if  any  of  the 
threat  systems  cannot  be  fully  addressed  in  testing,  the 
limitations,  as  well  as  the  testers'  plan  to  compensate  for  the 
limitations,  must  be  included  in  the  TEMP.  The  test-threat 
limitations  must  be  addressed  in  sufficient  detail  to  provide  an 
understanding  of  their  impact  on  the  test  and  thereby  the  impact 
on  providing  data  and  information  with  which  to  address  the 
issues  and  criteria. 

d.  lER  and  TER  will  address  the  approved  threat  of  the 
requirements  document,  as  well  as  the  threat  projected  to  exist 
post-IOC  as  described  in  the  STAR.  lER  and  TER  will  separately 
address  each  element  of  the  approved  threat,  as  well  as  the 
approved  threat  in  existence  at  the  last  milestone  review,  if 
different. 

e.  Actual  threat  systems  will  be  used  as  targets  or 
simulators  during  testing.  When  such  threat  systems  are  not 
available,  simulators  or  threat  systems  that  have  been  accredited 
via  the  Validation  and  Accreditation  Plan  for  Threat  Simulators 
and  Targets  will  be  used.  Requirements  for  threat  systems, 
simulators,  and  targets  are  to  be  coordinated  with  the  PM,  ITTS 
(STRICOM) . 

f.  Test  planning  must  reflect  the  threat  against  a  supporting 
system  or  a  system  interoperating  with  the  system  under  test 
(such  as  a  computer  system  dependent  on  a  separate  coitanunications 
system) . 

g.  Smoke  and  obscurants  and  laser  vulnerability  will  be 
addressed  as  a  part  of  all  threat  considerations  for 
electromagnetic  and  optical  systems. 

13-5.  Threat  Challenge 
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Meeting  this  objective,  particularly 

intelligence  assessments  into  instrumented  field  test  arrays 
adequate  to  test  a  developmental  system  within  the  context  of  th 
operational  issues  and  criteria  (OIC) ,  exit  criteria,  and 
technical  characteristics,  is  one  of  the  more  demanding 
chaninges  confronting  testers  and  evaluators  Given  resource 
constraints  that  preclude  representation  of  a  threat 
complete  fidelity,  testers  and  evaluators  must  be  persistent  an 
resourceful  in  seeking  means  to  offset  threat  portrayal 
shortfalls  to-minimize  their  impacts  as  potential  test 
limitations  with  emphasis  on  those  aspects  directly  related  to 

the  OIC. 

13-6.  Modeling  and  Simulation  of  the  Threat  ,,  v. 

Application  of  modeling  and  simulation  techniques  should  be 
considered  as  a  means  to  offset  the  impacts  of  test  limitations 
and  assess  the  impacts  of  uncertainties  that  exist  in  the  threat 
data  used  in  the  test. 


13-7.  Threat  Aspects 

a.  Current  and  projected  military  capabilities  of  a  potential 

enemy  to  limit,  neutralize,  or  destroy  ® 

mission,  organization,  or  item  of  equipment  under  battlefield 

conditions  at  a  post-IOC  date. 


b.  Intended  targets  of  the  U.S.  developmental  system. 
0,  Probable  enemy  reactive  threats  to  the  system. 


Section  II 

Derivation  of  Threat 


13-8.  Threat  intelligence 

This  term  pertains  to  intelligence  that  describes  the  current  and 
projected  future  capabilities  of  a  potential  enemy  force  against 
one  or  more  U.S.  developmental  systems  in  terms  of  combat 
materiel,  employment,  doctrine,  force  structure,  and  combat 
environment,  tailored  to  support  specific  combat 

development  projects.  It  integrates  pertinent  data  from  general, 
scientific,  technical,  and  estimative  intelligence  disciplines. 
Threat  intelligence  is  derived  from  DA-approved  and  validated 
intelligence  products  which  include  studies  as  well  as  models, 
simulations,  and  scenarios.  A  family  of  threat  documents  define, 
with  progressively  greater  specificity,  the  threats  for 
deve lopmenta 1  sy stems . 
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13-9.  Military  Capabilities  Studies  (MCS) 

These  studies  replace  the  Soviet  Battlefield  Development  Plan 
and  will  serve  as  a  threat  counterpart  to  the  TRADOC  Battlefield 
Development  Plan  (BDP)  and  as  Army  baseline  intelligence  on 
various  threat  forces  in  support  of  the  Army  acquisition  process. 
They  are  produced  by  Army  Intelligence  and  Security  Command 
(INSCOM)  and  Defense  Intelligence  Agency  (DIA)  production  centers 
and  approved  by  DA  DCSINT/DIA. 

13-10.  System  Threat  Assessment  Report/System  Threat  Assessment 
(STAR/STA) 

a.  The  STAR/STA  is  the  basic  threat  document  supporting 
system  development  for  both  major  and  nonmajor  acquisition 
programs.  It  is  used  to  define  the  threat  environment  in  which  a 
developmental  system  must  function  throughout  its  life  cycle, 
typically  at  the  IOC  date  plus  1,0  years. 

b.  The  STAR/STA  supports  Army  Force,  Combat,  and  Material 
Development.  The  STAR/STA  shall  be  written,  approved,  and 
updated  continuously  throughout  the  system  development  life  cycle 
(DoDD  5000.1) . 

c.  STAR/STA  are  derived  from  regional  or  country  MCSs  and 
appropriate  Army/DIA  scientific  and  technological  intelligence 
(S&TI)  products  from  the  DIA  and  INSCOM  analytical  organizations. 

d.  Although  both  documents  have  identical  formats,  they  have 
differing  approval  and  validation  requirements  since  the  STAR  is 
produced  to  support  ACAT  I  and  II  and  nonmajor  programs 
designated  for  OSD  T&E  oversight,  while  the  STA  pertains  to  other 
nonmajor  ACAT  III  and  IV  systems. 

13-11.  Threat  Test  Support  Package  (TTSP) 

a.  The  Threat  TSP  is  derived  from  the  STAR/STA,  but  is  more 
detailed  and  provides  the  threat  scenarios  to  support  a  specific 
test  and  assesses  the  impacts  of  threat-related  test  limitations. 

b.  For  ACAT  I,  ACAT  II,  and  other  OSD  T&E  oversight  programs, 
the  Office  of  the  DCSI  (ODCSI)  will  approve  the  TTSP.  The  TTSP 
for  all  other  programs  will  be  approved  by  the  TRADOC  Combined 
Arms  Command,  Threat  Directorate  for  OT,  and  the  USAMC,  ODCSI, 
for  DT.  The  threat  manager  for  the  appropriate  TRADOC  center  or 
school  is  the  approval  authority  for  the  TTSP  for  CEP. 

13-12.  Critical  Intelligence  Parameters  (CIP) 
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a.  CIPs  serve  as  a  mechanism  for  determining  when  an 
evolving  "threat”  may  impact  on  the  U.S.  developmental  system. 

b.  Together,  the  PM  and  the  AMC  senior  intelligence  officer 
(SIO)  of  the  proponent  centers  and  commands  establish  limits 
defining  the  degree  that  threat  can  change  without  causing  a 
major  redesign  or  reassessment  of  the  program.  These  threat 
limits,  expressed  as  CIPs,  define  thresholds  which,  if  breached, 
could  change  the  operational  requirement  for  a  system 
significantly. 

c.  Once  defined,  CIPs  are  submitted  through  intelligence 
channels  for  DA  validation.  CIPs  serve  to  alert  supporting 
intelligence  analysts  regarding  specific  priority  intelligence 
requirements  as  well  as  justify  subsequent  intelligence^ 
production  efforts  and,  if  needed,  supplemental  collection 
actions. 


Section  III 

Targets  and  Threat  Simulators 


13-13.  Use  of  Threat  Simulators  and  Targets 

Whenever  possible,  actual  threat  systems  are  used  during  OTE  to 
represent  an  enemy  force,  but  resource  limitations  usually  result 
in  the  use  of  replicas,  threat  simulators,  and  surrogates,  the 
functional  characteristics  of  which  approximate  those  of  actual 
threat  systems.  Threat  simulators  generally  are  more  costly  and 
sophisticated  than  targets  and  are  intended  for  reuse,  while 
targets  are  devices  which  are  designed  to  be  engaged  and 
destroyed . 

13-14.  Program  Manager,  Instrumentation,  Targets,  and  Threat 
Simulators  (PM,  ITTS) 

The  PM,  ITTS,  a  subordinate  of  STRICOM,  is  responsible  for  the 
engineering,  development,  acquisition,  fielding  and  capability 
accounting  of  Army  targets,  threat  simulators,  and  major  range 
instrumentation  for  DT  and  OT.  It  is  the  executive  agent  for 
both  the  ATS  and  Army  Targets  Programs. 

13-15.  ATS  Program 

With  transfer  of  US  Army  Missile  and  Space  Intelligence  Center 
(MSIC)  from  the  former  Army  Intelligence  Agency  to  DIA  and  the 
transfer  of  the  Threat  Simulator  Management  Office  (formerly 
Threat  Simulator  Project  Office)  from  MSIC  to  PM,  ITTS,  the 
latter  organization  became  the  materiel  developer  for  the  ATS 
Program.  The  Threat  Simulator  Management  Office  (TSMO)  plans. 
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organizes,  directs,  and  manages  the  ATS  Program.  The  Army  FSTC, 
supports  TSMO  in  developing  nonmissile-related  threat  simulators. 

13-16.  Army  Target  Program 

Targets  Management  Office  (TMO) ,  upon  transfer  from  MICOM  to  PM, 
ITTS  became  the  materiel  developer  for  Army  ground  and  aerial 
targets  for  use  in  DT  and  OT. 

13-17.  Threat  Simulator  and  Target  Validation 

a.  Validation  is  the  process  used  to  determine  whether  a 
simulator  or  target  provides  a  sufficiently  realistic 
representation  of  a  corresponding  threat  system  to  justify 
continuation  of  its  development,  use,  or  modification  to  restore 
or  improve  its  capabilities  to  conform  with  current  intelligence 
estimates. 

b.  Under  provisions  of  the  U.S.  Army  Validation  and 
Accreditation  Plan  for  Threat  Simulators  and  Targets,  validation 
working  groups  (VWG) ,  comprised  of  representatives  of  user, 
intelligence,  and  simulator/ target  development  organizations  are 
convened  by  TEMA,  which  charters  them  and  designates  the  chairman 
(normally  an  analyst  from  the  appropriate  S&TI  production 
center) .  PM,  ITTS  determines  when  VWGs  are  required,  informs 
TEMA,  and  also  participates  in  the  meetings. 

c.  Validation  is  performed  at  key  decision  points  during  the 
life  cycle  of  simulator  or  target;  Design  specification  review; 
critical  design  review;  IOC  (acceptance);  and  operational  (upon 
major  modification  and  periodically  following  acceptance  for  use 
in  testing. 

d.  The  DoD  Executive  Committee  on  Threat  Simulators  is 
notified  of  all  VWG  actions  and  it  must  approve  the  design 
specification  review  results  prior  to  the  award  of  a  contract  to 
develop  a  simulator  and  the  results  of  acceptance  VWG  before  the 
simulator  can  be  used  in  testing. 

13-18.  Technical  and  Use  Analyses 

During  validation,  a  technical  analysis  (a  comparison  of 
simulator  capabilities  with  current  S&TI)  and  a  use  analysis  (a 
comparison  of  simulator  capabilities  and  limitations  with  the 
projected  general  use  or  known  requirements)  are  evaluated  to 
determine  the  validity  for  its  intended  role  in  testing. 

13-19.  Foreign  Materiel  Program  (FMP) 

DA  DCSINT  has  overall  Army  staff  responsibility  for  the  FMP  for 
the  acquisition  and  exploitation  of  foreign  materiel,  which  is  a 
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source  of  actual  equipment  for  use  in  testing  developmental  U.S. 
systems.  The  acquisition  and  exploitation  functions  are  executed 
by  INSCOM,  and  FSTC,  respectively. 

13-20.  OPTEC  Threat  Support  Activity  (OTSA) 

OTSA  operates  and  maintains  the  threat  simulators  in  its 
inventory  in  support  of  Service  testing  and  training  activities. 
It  assists  in  test  planning,  and  participates  in  VWGs,  threat 
accreditation  working  groups,  and  OTRR. 


Section  IV  .  ,  ^  ^4. 

Threat  Support  to  Force,  Combat,  and  Materiel  Development 


13-21.  Guidance  (AR  381-11) 

a.  As  the  principal  source  of  guidance  for  threat  ^ 
intelligence  support,  AR  381-11  reaffirms  the  guidance  in  DoD 
Instruction  5000.2  that  the  objective  of  threat  support  is  to 
ensure  that  each  system  developed  is  mission  capable  in  its 
intended  operational  environment  throughout  its  expected 
operational  life. 

b.  DA  DCSINT  is  an  approval/validation  authority  for  threat 
documents  produced  by  combat  developers  and  materiel  developers, 
to  support  acquisition  programs  for  materiel  systems  and  ensures 
that  approved  threat  is  used  in  testing. 

c  Threat  to  be  portrayed  in  testing  for  major  programs  (ACAT 
I  and  II)  and  nonmajor  (ACAT  III  and  IV)  programs  designated  for 
OSD  T&E  oversight,  must  be  based  on  DIA  validated  threat  data 
sources  and  is  subject  to  DIA  guidance,  review,  or  validation. 

For  other  ACAT  III  and  IV  programs,  the  appropriate  MACOM  is 
responsible  for  coordinating  the  preparation  of  threat 
assessments  and  threat  test  support  documentation. 

d  TRADOC  is  the  principal  combat  developer  for  material 
systems  and  tactical  automated  information  systems  (AIS)  with 
INSCOM,  Hospital  Service  Command,  Medical  Research  Command,  Corps 
of  Engineers,  Criminal  Investigation  Command,  and  the  Army 
Strategic  Defense  Command  also  designated  by  AR  70-1  as  combat 
developers  for  specialized  materiel. 

e.  AMC  is  the  principal  materiel  developer  for  materiel 
systems.  The  Information  Systems  Command  has  both  combat  and 
materiel  developer  responsibilities  for  sustaining  base  and 
strategic  automated  information  systems. 
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f .  These  MACOMs  prepare  and  approve  the  threat  inputs  for 
program  management  documents  and,  when  required  to  support 
testing,  the  necessary  threat  documentation  as  well  as  validating 
threat  portrayals. 

13-22.  Program  Management  Documents 

The  threat  portions  of  system  programmatic  documents  for  major 
programs  and  nonmajor  programs  designated  for  OSD  T&E  oversight 
are  prepared  by  the  AMC  DCSINT  and  TRADOC  Combined  Arms  Command 
(CAC)  Threat  Directorate  (TD) ,  working  cooperatively.  These 
include  the  MNS,  ORD,  integrated  program  summary  (IPS),  and  COEA, 
which  support  the  decision  to  enter  EMD  for  the  system.  These 
threats  are  validated  by  DA  DCSINT.  For  other  nonmajor  programs, 
CAC/TD  and  AMC  DCSINT  prepare  and  approve  the  threat  portions  of 
these  programmatic  documents. 

13-23.  Test  Support  Documents 

Because  a  number  of  organizations  share  responsibility  for  the 
complex  and  demanding  task  of  integrating  threat  into  T&E,  the 
major  provisions  of  AR  381-11  and  related  regulations  (TRADOC 
Regulation  381-1  and  Pamphlet  381-3)  should  be  consulted  for 
detailed  explanations  of  organizational  responsibilities  with 
respect  to  threat  support.  The  process  of  integrating  threat 
into  T&E  programs  requires  that  DCSOPS,  DCSINT,  AMC,  TRADOC, 
TECOM,  AMSAA,  and  OPTEC  coordinate  closely  and  constantly 
throughout  the  acquisition  process. 


Section  V 

Required  Characteristics  of  Threat  Support  to  T&E 


13-24.  Consistency 

The  threat  environments  applied  to  testing  of  developmental 
systems  must  be  derived  from  a  baseline  of  DA— approved 
intelligence  products.  Threat  portrayals  for  DT  and  OT  of  a 
system,  while  tailored  for  each  test,  must  remain  compatible 
throughout  testing. 

13-25.  Continuity 

The  planned  portrayal  of  threat  must  be  evaluated  at  each  phase 
in  the  T&E  cycle  to  ensure  that  related  shortfalls  are  identified 
in  T&E  documents  as  test  limitations  and  their  impacts  on  the 
validity  of  the  test  are  assessed.  Efforts  to  incorporate  the 
most  current  threat  intelligence  in  test  planning  and  to  upgrade 
the  fidelity  of  planned  threat  portrayals  must  be  continuous. 

13-26.  Timeliness 
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Intelligence  estimates  of  the  threat,  even  though  they  may  treat 
specific  aspects  of  future  threat  forces  capabilities  with 
uncertainty  due  to  intelligence  "gaps”,  must  be  provided  to 
developers  and  testers,  on  a  timely  basis,  to  meet  prescribed 
planning  milestones  throughout  the  T&E  cycle. 


Threat  must  be  tailored  to  each  test  to  ensure  that  the  simulated 
battlefield  environment  is  adequate  to  test  the  developmental 
system  in  the  context  of  the  OIC  it  must  satisfy.  In  defining 
the  threat  for  developers,  testers,  and  evaluators,  implications 
of  incomplete  intelligence  must  be  identified  to  them  in  terms  of 
"gaps"  and  uncertainties  to  allow  early  consideration  of  the 
application  of  automated  modeling  and  simulation  technigues^ 
necessary  to  integrate  relevant  threat  intelligence  uncertainties 
into  T&E  processes. 

13-28.  Comprehensiveness  ,  .  . 

The  threat  against  the  total  system  must  be  described  and  incluae 
supporting  systems  or  other  interoperating  systems,  such  as  a 
computer  system  dependent  on  a  separate  communications  system. 


Section  VI 

Management  of  the  Threat  during  Test  Planning 


13-29.  Planning  Overview  ^  ^  4.v, 

Operational  testers  and  evaluators  are  expected  to  understand  the 
evolving  threat  and  integrate  it  into  operational  tests  that 
address  COIC  or  exit  criteria,  AOIC,  or  technical  characteristics 
and  are  realistic,  representative,  and  credible.  Threat-related 
issues  should  be  managed  using  the  following  guidelines: 

13-30.  Planning  Coordination  for  OT 

a.  Coordination  between  testers  and  evaluators  with  the 
appropriate  MACOM  threat  support  organization  (usually  the  T^DOC 
center  or  school  threat  manager)  responsible  for  the  production 
of  the  STAR/STA  and  TTSP  should  be  established  early  and  continue 
throughout  test  planning. 

b.  In  addition  to  approved  COIC  or  exit  criteria,  the 
supporting  threat  organization  roust  have  access  to  the  AOIC  and 
the  planning  data  embodied  in  the  operational  test  design  concept 
(OTDC)  in  Chapter  2  of  the  test  and  evaluation  plan  (TEP) .  The 
OTDC  includes  the  scope  (tactical  scenarios,  degree  of 
operational  realism,  and  types  of  test  events),  test  factors  and 
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conditions  (control  of  factors  to  ensure  test  events  occur  under 
appropriate  combinations  of  test  conditions) ,  and  test  design 
matrices  (grouping  of  test  conditions  into  trials,  vignettes, 
missions,  and  phases) . 

c.  Since  the  TTSP  supports  preparation  of  the  test  design 
plan  and  must  be  prepared  to  meet  regulation-specified  OTE 
planning  timelines,  the  supporting  threat  organization  must 
receive  OTDC  data  as  early  as  possible  (see  Part  V  on  OTE 
planning  processes) . 

13-31.  Review  of  Threat  Documents 

Threat  documents,  the  STAR/STA,  its  updates,  and  TTSP,  are  of 
concern  to  the  evaluator,  who  is  responsible  to  review  them  for 
adequacy  and  to  ensure  they  have  been  approved  and  validated  lAW 
Table  2-1,  AR  381-11,  before  use. 

13-32.  Ad-hoc  Intelligence  Integrating  Working  Groups 

Figure  13-1  is  a  simplified  schematic  of  the  relationship 
of  supporting  threat  intelligence  documents  and  the  OTE  process. 
The  chart  illustrates  that  the  flow  of  threat  support  is  a 
complex  process  with  multiple  layers  of  review  for  approval  and 
validation  and  there  are  few  interfaces  between  threat  support 
and  T&E  processes.  Consequently,  testers  and  evaluators  must 
actively  participate  in  several  "ad-hoc”  intelligence  working 
groups  for  integrating  the  threat  support  provided  to  T&E  and 
test  readiness  reviews  to  ensure  the  readiness  of  plans  and 
resources,  which  includes  those  related  to  the  threat. 

(INSERT  FIGURE  13-1) 

b.  DA  DCSINT  designates  a  threat  integration  staff  officer 
(TISO)  for  specific  mission  areas  to  coordinate  closely  with 
representatives  of  the  HQ  DA  staff,  combat  and  materiel 
developers,  OPTEC,  program  executive  offices  (PEO)  and  the  Army 
intelligence  community  to  assure  responsive  threat  support  and 
guidance  throughout  the  life  cycle  of  a  program.  The  TISO 
accomplishes  much  of  this  coordination  through  "ad  hoc”  threat 
integrating  bodies. 

13-33.  System-Specific  Threat  Coordinating  Group  (TCG) 

The  threat  integrator  member  of  the  TIWG  will  establish  and  chair 
a  TCG  as  a  subgroup  of  the  TIWG.  For  major  and  designated 
nonmajor  materiel  programs,  this  will  be  accomplished  by  the  DA 
DCSINT  TISO,  while  the  TRADOC  TM/AMC  FIO,  as  appropriate,  has 
this  responsibility  for  other  nonmajor  programs.  The  TCG 
includes  representatives  from  TRADOC  CAC/TD,  the  center  or 
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school  TM,  AMC  DCSINT,  OPTEC  (tester  and  evaluator),  and  the  DIA 
and  INSCOM  intelligence  producing  organizations  (as  required) . 

The  TCG  operates  throughout  the  life  cycle  of  the  developmental 
system.  The  System-Specific  TCG  performs  the  following 
functions: 

a.  Coordinate  the  production  and  approval/validation  of 
threat  intelligence  in  support  of  Army  developmental  systems. 

b.  Review  and  coordinate  approval  of  STARS  and  TTSPs  and 
threat  portions  of  system  program  management  documents. 

c.  Review  combat  simulations  and  models  and  intelligence  data 
for  correct  threat  applications. 

d.  Review  and  coordinate  threat  support  to  testing  to  include 
scenarios  and  use  of  simulators,  surrogates,  and  targets. 

e.  Assist  combat  and  materiel  developers  to  identify  and 
articulate  their  intelligence  requirements,  to  include  those 
responding  to  CIP  requirements,  and  facilitate  preparation  of 
appropriate  production  tasking  for  the  intelligence  community. 

f.  Identify  threat  and/or  threat  support  issues  and  determine 
responsibility  for  resolution. 

13-34.  Threat  Accreditation  Working  Group  (TAWG) 

Under  provisions  of  the  U.S.  Army  Validation  and  Accreditation 
Plan  for  Threat  Simulators  and  Targets,  the  TAWG  operates  to 
approve  these  resources  for  specific  test  applications.  Included 
in  its  membership  are  representatives  from  the  same  organizations 
that  comprise  the  TCG  as  well  as  the  PM  ITTS,  threat  simulator 
and  target  materiel  developer  offices,  appropriate  S&TI  center 
analyst(s),  and  the  system  program  manager.  The  TAWG  should 
meet  at  least  24  months  prior  to  the  test  (T-720)  to  accomplish 
the  following  functions: 

a.  Accredit  the  use  of  designated  threat  simulators/  targets 
for  each  test. 

b.  Ensure  that  threat  resources  in  the  final  outline  test 
plan  (OTP)  are  adequate  to  represent  the  threat  prior  to 
submission  to  the  Test  Schedule  and  Review  Committee  (TSARC) . 

c.  Identify  differences  ("deltas”)  between  the  simulators  or 
targets  and  current  estimates  of  corresponding  threat  system 
characteristics  and  assess  their  impacts  on  the  test. 
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d.  Through  comparison  of  the  drafts  of  the  TTSP  and  Chapter  2 
of  the  TEP,  accreditation  offers  a  timely  opportunity  to 
reconcile  differences  between  them.  Also,  this  facilitates 
development  of  test  planning  guidance  as  a  basis  to  complete 
Chapter  3  of  the  TEP  and  provide  increased  assurance  that  the 
threat  resources  identified  in  the  OTP  will  be  sufficient  to 
represent  the  threat  with  greater  fidelity  during  the  test. 

13-35.  Operational  Test  Readiness  Reviews  (OTRR) 

TRADOC  CAC/TD  is  responsible  to  validate  the  planned  threat 
portrayal.  For  force-on-force  tests,  it  also  validates  the 
threat  force  training  plan  prepared  by  the  TM.  This  validation 
is  documented  in  the  Operational  Test  Readiness  Statement  (OTRS) 
prepared  by  the  combat  developer.  OTSA  also  participates  to 
report  of  the  preparedness  of  threat  simulators. 

13-36.  Management  of  Threat-Portrayal  Shortfalls 
Testers  and  evaluators,  through  coordination  and  active 
participation  in  the  TCG  and  TAWG  will  be  able  to  identify 
potential  threat  portrayal  shortfalls  in  early  in  T&E  planning. 

13-37.  Deviations  From  the  Threat 

When  significant  deviations  from  the  validated  threat  are 
expected  in  test  portrayals,  whether  due  to  a  lack  of  threat 
resources  or  situations  dictated  by  testing  requirements  and/or 
it  is  determined  that  potential  portrayal  shortfalls  pose 
significant  risks  to  test  validity,  the  appropriate  TM  and  threat 
integration  center  (usually  CAC/TD)  should  be  consulted  so  they 
can  seek,  '•offsets'*  or  alternatives  to  minimize  potential  threat- 
related  test  limitations.  If  necessary,  CAC/TD  can  seek  formal 
DA  DCSINT  TISO  recommendations  for  alternative  solutions  to 
permit  early  resolution  of  problems  by  OTE  decision  makers. 

13-38.  Threat  Uncertainties 

When  significant  uncertainties  exist  in  supporting  threat 
intelligence,  or  there  are  significant  threat  portrayal 
shortfalls,  the  use  of  modeling  and  simulation  techniques  as  a 
means  of  compensating  for  these  limitations  should  be  considered. 


Section  VII 

Management  of  On-Site  Threat  Portrayal  Issues 


13-39.  Threat  Portrayal  Fidelity 

a.  Due  to  resource  limitations,  it  is  unlikely  that  the 
threat  force  in  a  test  will  be  represented  with  total  fidelity  to 
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the  threat  as  described  in  the  STAR  and  specifically  defined  for 
the  test  in  the  TTSP.  The  degree  to  which  threat  force 
operations  will  be  faithfully  represented  during  the  test  will  be 
based  on  subjective  judgements  of  those  who  witness  it. 

b.  For  T&E  purposes,  the  true  criteria  for  this  judgement  is 
the  adequacy  of  the  threat  portrayed  to  support  realistic  testing 
of  the  developmental  system  within  the  context  of  the  COIC  or 
exit  criteria  and  AOIC  it  must  satisfy  rather  than  a  strict 
comparison  of  the  portrayed  threat  point-by-point  against  the 
"text  book". 

13-40.  Threat  Critiques 

Intelligence  personnel  supporting  or  observing  test  preparations 
and/or  execution  should  direct  commentary  or  critiques  on  the 
threat  portrayal  through  the  evaluator.  This  will  ensure  that 
only  those  comments  deemed  relevant  to  the  interpretation  and 
evaluation  of  test  results  are  communicated  to  other  personnel 
directly  associated  with  the  test. 

13-41.  Resolution  of  Threat  Shortfalls 

Normally,  the  TRADOC  center  or  school  TM,  who  is  responsible  for 
the  STAR/STA  and  TTSP  and  assisting  in  setting-up  the  test  and 
overseeing  its  threat— related  aspects,  and  the  CAC/TD,  the  Army 
validating  authority  for  threat  portrayals,  will  be  on-site  and 
are  capable  of  interpreting  the  significance  of  threat-related 
issues  on  test  validity,  thereby  minimizing  the  potential  for 
controversy. 

13-42.  Threat  Limitations  on  Test 

Significant  portrayal  shortfalls  must  be  included  in  OTE  reports 
as  "test  limitations"  and  their  impact  on  test  validity  assessed 
in  test  and  evaluation  reports. 


Section  VIII 

Threat  Integration  in  Test  Planning 


13-43.  Significance  of  Threat  for  T&E 

The  "threat"  is  an  essential  test  element  which  is  difficult  to 
represent  and  typically  has  the  greatest  impact  on  the  overall 
results.  Testers  and  evaluators  are  required  to  identify  the 
appropriate  threat  from  the  STAR/STA,  the  means  for  replicating 
that  threat  in  testing,  and  the  methods  for  quantifying  its 
effect  on  the  test  and  the  system. 

13-44.  Threat  Challenges 
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a.  OTE  is  based  on  COIC  developed  early  in  the  program  and 
AOIC,  which  provide  testable  criteria  to  support  evaluation  of  a 
related  COIC,  that  must  be  answered  to  support  a  production 
decision.  Answers  to  many  of  these  OIC  are  ''threat  sensitive"; 
that  is,  they  depend  heavily  on  the  threat  environment  to  which 
the  system  is  subjected  in  testing. 

b.  The  challenge  confronting  testers  and  evaluators  is  to 
ensure  that  the  threat  is  current,  realistic,  and  sufficiently 
stressful  to  adequately  test  the  system  against  the  COIC  or  exit 
criteria  it  must  satisfy  and  that  it  is  faithfully  replicated  in 
the  test.  Several  factors  complicate  these  tasks: 

13-45.  Threat  is  Dynamic  and  Uncertain 

The  "threat"  to  be  portrayed  in  testing  results  from  an 
intelligence  estimative  analytical  process  which  assesses 
specific  military  capabilities  of  a  potential  enemy  usually  at 
future  points  in  time.  Although  "uncertainty"  is  inherent  in  all 
intelligence,  estimative  intelligence,  due  to  the  limited 
availability  of  collectable  information,  to  a  greater  degree  than 
other  types  of  analytical  disciplines,  is  heavily  reliant  on 
applied  methodologies  usually  derived  from  the  physical  sciences. 
As  new  intelligence  is  developed  and  intelligence  "gaps"  narrow 
or  close  as  a  result  of  supplemental  collection  and  analysis  or 
evolving  methodologies,  the  threat  may  change.  If  the  DA  threat 
coordinating  group  determines  these  changes  to  be  substantial, 
they  must  be  incorporated  into  T&E  activities. 

13-46.  Threat  Inconsistency 

The  COIC,  defining  acceptable  standards  of  system  performance, 
are  formulated  before  the  STAR/STA.  As  a  result,  there  may  be 
differences  between  the  threat  outlined  in  the  STAR/STA  and  the 
threat  considered  in  developing  the  COIC/AOIC.  The  situation 
also  can  arise  with  the  TTSP  which  may  require  modification  to 
accommodate  evolving  OIC  or  exit  criteria  and  test  planning. 

13-47.  Threat  Scenarios 

a.  Defense  Guidance  (DG) .  The  annual  DG,  issued  by  the 
Secretary  of  Defense,  provides  a  set  of  common  planning 
assumptions  for  U.S.  and  friendly  forces  and  planning  scenarios 
projected  for  a  ten-year  period.  It  also  defines  strategy  and 
force  options  identifying  the  specific  operational  environments 
in  which  U.S.  forces  must  be  prepared  to  function.  The  DG  is 
also  the  basis  for  development  of  U.S.  Army  scenarios  to  support 
the  force  and  materiel  development  processes. 

b.  TRADOC  Standard  Scenarios.  The  purpose  of  a  standard 
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scenario  is  to  provide  consistency  and  reduce  bias  for  all  combat 
development  programs  through  use  of  a  common  base  case  that 
portrays  TRADOC-approved  U.S.  Army  doctrinal  and  operational 
concepts.  The  TRADOC  Analysis  Command  is  the  proponent  for 
scenario  development  for  friendly  forces,  while  TRADOC  CAC/TD 
assists  in  preparation  of  the  threat  force  scenario,  which  is 
validated  by  DA  DCSINT.  TRADOC  standard  scenarios  are  considered 
in  the  development  of  threat  force  scenarios  in  the  Threat  TSP 
and  preparation  of  the  Integrated  Threat  Tactical  Operations 
Plan,  both  of  which  support  the  test  design  process.  During  OTP 
preparation/preliminary  test  design  planning,  the  system 
proponent  and  the  operational  tester,  subject  to  TIWG  approval, 
select  the  standard  scenario  for  use  in  testing.  Both  friendly 
and  threat  test  operations  must  be  compatible  with  the  selected 
standard  scenario. 

c.  Integrated  Threat  Tactical  Operations  Plan  (ITTOP) .  The 
ITTOP  is  an  instructional  guide  for  the  operation  of  simulators 
also  useful  in  test  planning,  specifically  as  a  reference  in 
preparing  both  Chapter  3  of  the  TEP  and  the  detailed  test  plan 
(DTP) .  It  is  produced  by  OTSA,  approved  by  OPTEC,  and  validated 
by  DA. 


Section  IX 

Relationship  of  Threat  Documents  to  OTE 


13-48.  General  Relationships 

The  STAR/STA  as  used  in  support  of  OTE  defines  the  threat 
environment  in  which  an  emerging  system  must  function  throughout 
its  life  cycle,  while  the  TTSP  expands  the  content  of  the 
STAR/STA  in  sufficient  detail  for  use  in  defining  the  environment 
necessary  to  plan  and  conduct  the  OT. 

13-49.  STAR/STA  Preparation 

a .  Purpose . 

(1)  The  STAR/STA  is  the  basic  authoritative  system  threat 
assessment  tailored  for  and  focused  on  a  specific  defense 
acguisition  program.  It  is  used  as  a  basis  to  the  prepare  the 
threat  portions  of  system  programmatic  documents  to  include  the 
IPS,  ORD,  and  COEA,  which  support  the  decision  to  enter  EMD. 

(2)  For  OTE,  the  STAR/STA,  is  used  to  define  the  "tactical 
context"  to  support  development  of  the  TEMP,  outline  test  plan 
(OTP),  and  Chapters  1  and  2  of  the  TEP. 
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b.  Preparation  and  Approval. 

(1)  Preparation  of  the  initial  STAR/STA  by  the  CBTDEV, 
usually  the  TRADOC  proponent  school  TM,  in  coordination  with 
materiel  developer,  normally  the  AMC  proponent  FIO,  commences 
upon  program  initiation  with  approval  of  the  MNS. 

(2)  For  most  major  programs  and  nonmajor  programs  designated 
for  OSD  T&E  oversight,  the  STAR  is  reviewed  and  approved  by 
TRADOC  CAC/TD  and  AMC  DCSINT  and  forwarded  to  DA  DCSINT  for 
validation  within  150  days  after  MDR  0. 

(3)  Most  STA  for  nonmajor  programs  are  jointly  validated  by 
TRADOC  CAC/TD  and  AMC  DCSINT.  AMC  normally  prepares  STAR/STA 
updates  for  subsequent  MDRs.  The  DIA  validates  STAR  used  for 
MDRs,  or  when  validation  is  requested  by  DA  DCSINT. 

(4)  Per  DoD  Manual  5000. 2M,  preparation  of  the  STAR/STA  for 
systems  which  are  not  threat  dependent  may  be  waived,  upon 
request,  by  the  appropriate  milestone  decision  authority. 

c.  Content  and  Format.  STAR/STA  format  and  content  are 
detailed  in  DoD  Manual  5000. 2-M,  AR  381-11,  Appendix  C  of  TRADOC 
Pamphlet  381-3,  and  outlined  in  Figure  13-2. 

(INSERT  FIGURE  13-2) 

d.  Criteria  for  Use.  Before  use  in  T&E  planning  the  STAR/ 
STA  must  be  approved  and  validated  lAW  AR  381-11,  Table  2-1. 

13-50.  TTSP 

a.  Purpose. 

(1)  Derived  from  the  STAR/STA,  the  TTSP  is  more  detailed  and 
is  used  in  developing  the  ’’OTE  environment"  necessary  to  prepare 
Chapter  3  of  the  TEP  and  provides  the  threat  scenarios  for  each 
user  test.  Determination  of  the  threat  year  and  scenario 
selection  for  the  test  will  be  made  by  the  TIWG  upon  the 
recommendation  of  the  system  proponent  and  the  evaluation 
organization. 

(2)  The  TTSP  defines  the  threat  portion  of  a  realistic 
operational  test  environment  adequate  to  test  and  evaluate  the 
developmental  system  in  the  context  of  related  COIC  or  exit 
criteria  and  AOIC. 

b.  Preparation  and  Approval. 


13-16 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


(1)  For  OTE,  the  CBTDEV,  normally  the  TRADOC  proponent 
center/ school  TM,  prepares  the  TTSP  for  each  lOT,  18  months 
(T-540)  before  the  test  start  date.  For  other  tests  (FDTE,  EUTE, 
LUT,  or  FOT,  a  TTSP  will  be  prepared  unless  the  TIWG,  acting  upon 
the  recommendation  of  the  evaluation  organization,  determines 
that  a  validated  threat  portrayal  is  not  required  for  the  test. 
The  requirements  of  the  OIC,  OTDC,  and  TEMP  will  form  the  basis 
for  a  recommendation  to  waive  the  TTSP.  For  DT  no  TTSP  is 
prepared  unless  threat  force  operations  are  to  be  portrayed. 

(2)  For  user  testing  of  tactical  systems,  the  threat 
integration  center,  usually  the  CAC/TD,  approves/ validates  the 
TTSP,  from  a  "tester's  perspective",  to  ensure  that  threat 
operations  are  portrayed  accurately  and  consistently.  DA  DCSINT 
is  the  validation  authority  for  TTSPs  for  ACAT  I  and  II  and  ACAT 
III  and  IV  materiel  programs  on  the  OSD  T&E  oversight  list  and 
provides  a  copy  to  DIA  for  review  and  comment.  Most  TTSP  for 
user  testing  of  other  nonmajor  systems  are  approved  and  validated 
by  the  TRADOC  CAC/TD,  while  this  is  done  by  appropriate  AMC  FIO, 
when  a  TTSP  is  required  to  support  DT.  TTSP  must  be  approved  and 
validated  14  months  before  the  test  date  (T-420) . 

c.  Content  and  Format.  TTSP  format  and  content  are  detailed 
in  Appendix  C,  AR  381-11,  TRADOC  Pamphlet  381-3,  and  outlined  in 
Figure  13-3.  It  is  prepared  in  modular  format  to  facilitate  the 
updating  process  from  test  to  test  since  only  those  parts 
required  for  a  given  test  need  to  be  completed.  Section  III 
(Threat)  often  requires  revision,  since  the  AOIC  and  the  TEP 
continue  to  evolve. 

(INSERT  FIGURE  13-3) 

d.  Criteria  for  use.  The  TTSP  must  accurately  reflect  the 
threat  assessment  embodied  in  the  STAR  and  be  validated/  approved 
lAW  Table  2-1,  AR  381-11.  The  appendices  of  the  TTSP  are  closely 
aligned  to  elements  of  the  TEP  for  the  OT.  These  relationships 
are  shown  in  Figure  13-4. 

(INSERT  FIGURE  13-4) 


Section  X 

Integration  of  Threat  Data  in  OT  Planning 


13-51.  Threat  and  Evaluation  MOE  and  MOP 
Although  the  evaluator  has  access  to  threat  intelligence 
(STAR/STA)  shortly  after  program  initiation  that  is  used  to 
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define  the  "tactical  context"  for  the  test,  actual  integration  of 
the  threat  into  OTE  does  not  begin  until  after  completion  of  the 
functional  dendritics,  which  do  not  consider  the  threat.  The 
dendritics  for  each  system  are  used  to  define  system  functions 
and  subfunctions,  clarify  primary  measures  of  effectiveness 
(derived  from  the  COIC) ,  and  formulate  measures  of  performance 
and  data  requirements  necessary  for  OTE. 

13-52.  Threat  and  Test  Factors  and  Conditions 

Threat,  however,  becomes  operative  as  the  evaluator  endeavors  to 
identify  factors  (test  variables  likely  to  effect  test  event 
outcomes)  and  the  conditions  (discrete  aspects  of  a  factor,  or 
factors,  often  expressed  as  a  range  of  values,  capabilities,  or 
operational  modes) .  Threat  data,  such  as  the  types  and  echelon 
of  forces,  types  and  numbers  of  systems,  and  doctrine  and 
tactics,  which  determine  threat  force  movements  and  operations 
under  varying  situations,  become  factors  and  conditions  for 
purposes  of  developing  a  test  concept.  Once  these  determinations 
are  made,  usually  through  use  of  a  matrix  approach  keyed  to  each 
operational  issue  and  associated  criteria,  the  evaluator  then 
must  decide  how  each  factor  and  condition,  including  those 
related  to  the  threat,  will  be  controlled  during  testing,  i.e., 
"fixed",  "systematically  varied",  "tactically  varied",  or 
"uncontrolled"  (See  Part  5) . 

13-53.  Threat  and  the  Tactical  Context 

The  STAR/STA  is  used  to  define  the  tactical  context  describing 
the  threat  environment  and  threat  systems  that  will  exist  at  the 
IOC  date  and  throughout  the  life  cycle  of  the  developmental 
system.  The  evaluator  uses  the  STAR/STA  to  identify  the  tactical 
setting  as  well  as  develop  the  factors  and  conditions  to 
formulate  the  "test  approach"  section  of  Chapter  2  of  the  TEP  for 
use  of  the  tester  to  prepare  Chapter  3,  the  test  design  plan. 

The  evaluator  should  make  this  same  information  available  to  the 
appropriate  threat  support  office,  usually  the  TRADOC 
center/school  TM,  as  early  as  possible,  in  order  to  expedite 
preparation  of  the  Threat  TSP,  which  is  essential  to  development 
of  Chapter  3.  As  the  tester  refines  the  OTDC  or  "test  approach" 
guidance  developed  by  the  evaluator,  must  continue  coordination 
with  the  TM  to  ensure  timely  completion  of  a  Threat  TSP  tailored 
to  test  requirements. 

13-54.  Threat  and  the  OTE  Environment 

In  contrast  to  the  tactical  context,  the  OTE  environment,  or 
the  combat  situations  in  which  the  system  will  be  tested  at  a 
post-IOC  time  usually  selected  by  the  tester,  is  derived  from  the 
OTDC,  test  design  plan,  and  TTSP.  Relationships  between  systems 
and  factors  that  enhance  and  diminish  operational  effectiveness, 
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in  terms  of  the  tactical  context  and  the  OTE  environment,  are 
illustrated  by  Figure  13-5.  Both  enhancing  and  diminishing 
factors  are  included  in  the  OTE  environment.  Measuring  the 
cumulative  effects  of  these  factors  on  system  operational 
effectiveness  is  a  primary  objective  of  OT. 

(INSERT  FIGURE  13-5) 

a.  Enhancing  Factors.  The  organization,  tactics,  and 
doctrine  of  employment  are  integrated  so  that  operational 
effectiveness  of  the  system  is  enhanced. 

b.  Diminishing  Factors.  At  the  same  time,  the  operational 
effectiveness  of  a  system  is  subjected  to  diminishing  factors. 

The  chief  diminishing  factor  standing  between  the  system  and  the 
achievement  of  its  mission  is  the  enemy,  i.e.,  the  threat. 

Others  factors  include  the  effects  of  weather,  terrain,  and 
interference  from  other  systems. 

13-55.  Test  Profile  Sets 

TTSP  contain  threat  profiles,  system  profiles,  and  environmental 
profiles.  Test  designers  merge  threat,  system,  and  environmental 
profiles  into  "test  profile  sets,”  which  are  incorporated  into 
chapter  3  of  the  TEP  (test  design  plan) .  The  relationships  among 
test  threat,  system,  and  environmental  profiles  in  test  profile 
sets  are  illustrated  in  Figure  13-6. 

(INSERT  FIGURE  13-6) 

13-56.  Threat  Profiles 

The  TTSP  contains  individual  test  threat  profiles  consistent  with 
the  overall  test  objectives,  scenarios,  and  threat  resources  to 
be  used.  Threat  profiles  describe  the  types  of  threat  and  threat 
equipment  that  the  system  is  likely  to  encounter,  specific  threat 
effects  anticipated,  threat  tactics  and  countermeasures,  threat 
doctrine  and  employment  practices,  and  threat  organizations.  The 
operational  tester  uses  the  threat  profiles  to  develop  the  OTE 
environment  and  the  target  arrays  for  the  test. 

13-57.  Scoping  of  Test  Profiles 

Because  the  number  of  possible  test  profile  sets  is  so  large  and 
COIC  can  be  resolved  through  analytical  means  other  than  OTE,  it 
is  neither  economical  nor  desirable  to  develop  threat  profiles 
for  every  possible  profile  set.  Therefore,  the  tester  must 
monitor  the  preparation  of  the  TTSP  closely  to  ensure  that: 

a.  Threat  profiles  are  configured  appropriately  for  the 
environmental  conditions  and  means  of  employment  (tactics. 
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doctrine,  and  organization)  most  of  which  are  important  in  order 
to  respond  to  the  test  issues. 

b.  Threat  profiles  are  developed  only  for  those  aspects  of  a 
threat  profile  that  are  technically  possible,  operationally 
feasible,  and  realistic. 

13-58.  Profile  Complexity 

Because  the  TTSP  becomes  progressively  more  complex  during  the 
system  development  process,  test  threat  profiles  also  increase 
correspondingly  in  scope  and  complexity. 

a.  For  EUTE,  the  test  threat  profiles  focus  on  potential 
targets,  countermeasures,  and  opposing  weapons  at  the  single 
system  one-on-one  level. 

b.  For  lOTE,  the  test  threat  profiles,  depending  on  the 
developmental  system,  can  expand  to  include  opposing  forces  up  to 
the  battalion  level. 

c.  At  FOTE,  the  test  threat  profiles  include  an  updated 
configuration  of  potential  opposing  forces  at  all  levels. 

13-59.  System  Profiles 

System  profiles  define  the  tactical  applications  of  the  friendly 
system  as  derived  from  the  STAR  (U.S.  system  description  section 
provided  by  the  combat  developer).  See  Figure  13-6. 

13-60.  Environmental  Profiles 

These  profiles  define  the  terrain,  weather,  communications,  and 
transportation  infrastructures,  friendly  interference  (e.g., 
radio  frequency) ,  time  and  distance  separating  operating  forces 
from  their  support  structure,  and  other  non-threat  conditions 
under  which  the  test  is  to  be  conducted.  The  test  environmental 
profiles  are  drawn  from  the  system  requirements  documents  and 
supporting  analyses.  See  Figure  13-6. 

13-61.  Lethality  and  Survivability 

a.  Direct  Effect  Systems.  For  those  kinetic,  chemical,  and 
directed-energy  weapons  which  have  direct  impacts  against  the 
threat  force,  effectiveness  is  measured  in  terms  of  lethality  and 
survi vabi 1 i ty . 

b.  Indirect  Effect  Systems.  Other  types  of  systems  are 
designed  to  operate  indirectly  against  threat  systems  by 
enhancing  the  lethality  and/or  survivability  of  a  primary  system, 
e.g.,  improving  the  mobility,  C3,  or  intelligence  support  of  a 
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lethal  system.  While  the  operational  effectiveness  of  indirect 
systems  cannot  be  measured  by  the  direct  impact  they  have  on  the 
threat  force,  they  can  be  measured  by  the  extent  to  which  they 
either  multiply  the  lethality,  or  increase  the  survivability,  of 
a  primary  (direct  effect)  system. 

c.  Combined  Effect  Systems.  Some  indirect  systems  and 
subsystems,  i.e.,  communications  and  target  acquisition,  however, 
are  subject  to  both  lethal  and  nonlethal  EW  threats.  Although 
EUTE  may  isolate  and  emphasize  the  EW  threats  against  indirect 
systems,  ultimately  a  determination  must  be  made  whether  the 
indirect  system  measurably  contributes  to  the  operational 
effectiveness  of  either  specific  lethal  systems  or  combat  forces 
overall.  These  determinations  are  difficult  and  tenuous  if 
indirect  systems,  such  as  intelligence  systems,  are  evaluated 
against  the  threat  of  deception,  or  if  EW  systems  are  measured 
against  enemy  communications. 


13-62.  Threat  Adequacy 

a.  The  QIC  may  require  measurement  of  the  combined  impact  of 
the  factors  that  enhance  and  diminish  operational  effectiveness 
on  lethality  and  survivability  or  the  multiplying  effect  of  one 
system  on  the  lethality  and  survivability  of  another  system. 

When  either  circumstance  exists,  the  operational  tester  and 
evaluator  must  ensure  that  the  threat  portrayed  in  the  test  will 
be  sufficient  to  support  evaluation  of  direct  effect  system  as 
well  as  the  impacts  of  indirect  effect  systems. 

b.  Lacking  an  adequate  threat  portrayal  that  considers  both 
types  of  systems,  the  evaluator  will  be  unable  to  make  accurate 
assessments  of  system  operational  effectiveness.  Figure  13-7 
illustrates  the  relationship  among  these  factors  and  shows  the 
importance  of  the  tactical  context  and  test  environment  in 
determining  the  operational  effectiveness  of  a  system  in  terms 
of  its  lethality  and  survivability  in  the  presence  of  a  threat 
force. 

(INSERT  FIGURE  13-7) 


SECTION  XI 

Threat  and  Modeling  and  Simulation 


13-63.  Applications 

Modeling/simulation  (M/S)  can  be  a  valuable  adjunct  to  testing  to 
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provide  data  when  actual  field  testing  is  either  infeasible  or 
impractical  due  to  factors  of  cost;  test  time;  unsuitability  of 
maneuver  space,  terrain,  or  weather;  security  considerations; 
safety;  threat  portrayal  shortfalls;  restriction  on  use  of  the 
electromagnetic  spectrum;  and  limited  instrumentation  other  test 
resources.  The  determination  to  employ  simulations  techniques 
can  be  based  on  threat  considerations  such  as  the  following: 

a.  Threat-related  resource  limitations.  Estimated  threat 
capabilities  cannot  be  adequately  represented  due  to  a  lack  of 
threat  simulators /targets  and/or  threat  surrogates  which  match 
estimated  threat  capabilities. 

b.  Uncertainties  and  variables.  Modeling  and  simulation 
techniques  have  considerable  potential  for  improving  the  fidelity 
of  the  portrayal  of  threat  in  OTE  activities.  There  are 
significant  uncertainties  related  to  the  estimates  of  future 
threat  capabilities  which  should  be  carefully  considered  in  all 
OTE  activities.  Sensitivity  analyses,  using  M/S  techniques,  can 
be  applied  to  examine  the  impacts  of  incomplete  or  uncertain 
estimative  intelligence  on  testing.  In  addition,  M/S  can  assist 
in  projecting  the  implications  of  future  enemy  reactive  threat  to 
the  system  being  tested.  Typical  aspects  of  the  threat  that  lend 
themselves  to  M/S  techniques  include: 

(1)  System  performance  characteristics,  for  which  Army  S&TI 
production  agencies  develop  "best  estimates”,  that  normally 
become  the  basis  for  OTE  field  testing  design,  as  well  as  high 
and  low  parametric  values  as  a  means  of  "bounding”  the 
uncertainties . 

(2)  Variables  related  to  evolving  threat  forces  as  a  result 
of  materiel  upgrades,  organizational  changes,  and  modifications 
of  doctrine  and  tactics. 

(3)  Scenario-related  operational  options  involving  the  types 
of  combat  operations  being  portrayed,  e.g.,  main  attack  versus 
supporting  attacks,  or  offense  versus  defense. 

c.  Pretest  M/S  applications. 

(1)  Test  planning.  An  important  use  of  M/S  techniques  in 
test  planning  is  the  refinement  of  test  scenarios  and  data 
matrices  to  decide  which  elements  of  system  performance  should  be 
the  focus  of  OTE.  To  do  this,  the  modeling  and  simulations  used 
must  relate  the  operational  effectiveness  and  suitability  of  the 
system  in  a  realistic  scenario,  with  appropriate  force  levels 
using  situations  identified  in  the  operational  mode  summary/ 
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mission  profile  (OMS/MP) .  This  allows  the  evaluator  to  do 
sensitivity,  contingency,  and  functional  analyses  for  various 
technical  and  force  mix  assumptions. 

(2)  Determine  "deltas”.  There  is  a  perceiyed  need  in 
designing  tests  to  compare  (or  determine  the  differences  or 
"deltas")  between  the  performance  of  threat  simulators/ targets 
deployed  in  the  test  array  and  evolving  intelligence  estimates  of 
the  characteristics  and  capabilities  of  the  actual  threat 
system (s) . 

13-64.  Model-Test-Model  (MTM)  Concept 

a.  Purpose.  In  the  context  of  the  MTM  concept,  M/S 
techniques  augment  field  test  results  to  accomplish  one  or  more 
of  the  following  general  purposes: 

(1)  Extrapolation  or  expansion  of  the  results  of  reduced- 
scale  field  testing  to  determine  the  outcome/ impacts  on  larger 
forces. 

(2)  Extrapolation  of  test  results  across  differing  scenarios, 
terrain,  and  environments  and  extend  test  results  in  time. 

(3)  Comparison  of  multiple  repetitions  of  test  events. 

(4)  Conduct  sensitivity  analyses  to  address  a  myriad  of  "what 
if"  questions  some  of  which  relate  to  threat  variables. 

b.  The  process. 

(1)  Step  1.  Use  of  a  combat  model  to  refine  test  design, 
chec)c  execution  timing,  plan  location  of  support  equipment,  and 
predict  field  test  outcome. 

(2)  Step  2.  Execute  the  field  test  and  then  calibrate  the 
model  with  test  results. 

(3)  Step  3.  Use  model  to  augment  field  test  results  for  one 
or  more  of  the  purposes  listed  above. 

13-65.  Threat  Support  to  MTM 

Although  there  are  rigorous  verification,  validation,  and 
accreditation  procedures  for  the  application  of  M/S  techniques  in 
OTE,  an  essential  prerequisite  for  their  use  is  a  process  to 
ensure  that  threat  representations  and  usage  modeled  or  simulated 
are  consistent  with  approved  estimative  intelligence  through  Army 
and  Defense  Intelligence  Agency  validation  (See  chapter  16, 
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Modeling  and  Simulation  in  Support  of  Test  and  Evaluation) . 

a.  Approval/validation  of  threat  data.  The  threat 
represented  in  the  model  must  be  documented  and  traceable  to  an 
approved  and  validated  STAR  and  Threat  TSP,  or  to  automated 
threat  data  from  other  approved  Army  high-  and  low-resolution 
models.  The  threat  portions  of  M/S  developed  by  TRADOC  are 
approved  by  CAC/TD  and  validated  by  DA  DCSINT.  Threat  data  to  be 
used  in  M/S  applications,  however,  is  validated  by  CAC/TD. 
Deviations  from  threat  data  contained  in  DA  DCSINT  and  DIA 
approved  intelligence;  however,  must  be  fully  documented  and 
approved  by  DA  DCSINT  before  use. 

b.  Threat  requirements  for  sensitivity  analyses.  If  M/S  is 
appropriate  to  conduct  sensitivity  analyses  related  to 
uncertainties  in  the  "threat",  the  evaluator  will  require  a  range 
of  threat  alternatives  or  variables,  e.g.,  threat  force  weapons 
and  systems  parameters  and/or  doctrinal,  organizational,  or 
operational  options  derived  by  intelligence  analysts. 
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Figure  13-1.  Relationship  of  threat  support  documentation  and 
operational  testing  and  evaluation  for  tactical 
systems . 
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Figure  13-2.  Format  of  the  STAR/STA. 
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Figure  13-3.  Format  of  the  Threat  TSP. 


CORRESPONDANCES  BETWEEN  TTSP  APPENDICES  AND  THE  TEP 

Appendix  A.  Test  concept.  The  tester  prepares 
the  test  concept  (section  3.1  of  the  TEP)  based  on  Chapters 
1  and  2  of  the  TEP  prepared  by  the  evaluator.  It  will  describe, 
in  detail,  the  test  scope  and  criteria,  which  will  be  used  to 
define  the  required  threat  for  a  specific  test. 

Appendix  B.  Threat  scenario.  This  is  written  by  the  TM  and  based 
on  the  OTDC  provided  by  the  evaluator,  must  describe  the  test 
setting  from  the  threat  force  perspective.  The  test  scope  also 
defines  the  standard  scenario  to  be  used,  the 

desired  degree  of  realism,  and  the  types  of  test  events.  It  should 
summarize  how  the  test  will  be  conducted  in  terms  of  scheme  of 
maneuver,  organization  and  types  of  equipment,  tactics,  and 
supporting  fires  or  forces. 

Appendix  C.  Descriptions  of  the  trials/ test  runs/ vignettes.  These 
are  prepared  by  the  TM  and  derived  from  the  test  factors  and 
conditions  and  test  design  matrices  provided  by  the  evaluator, 
which  define  test  force  operations.  The  threat  forces,  equipment, 
and  operations  to  be  used  in  the  test  scenario  must  be  adequate  to 
provide  an  appropriate  "OT  environment"  needed  to  resolve  the  QIC 
or  exit  criteria. 

Appendix  D.  Firer/ target  matrix.  Item  level  data  requests, 
typically  probability  of  hit  (Ph)  and  probability  of  kill  (Pk) 
numbers  in  the  form  of  a  firer /target  matrix,  prepared  by  the 
tester  and  TM,  must  be  submitted  through  TRADOC  Analysis  Command  to 
TRADOC  CAC/TD  and  AMC  DCSINT  for  approval. 

Appendix  E.  The  technical  deficiencies  of  scheduled  threat 
simulators,  targets,  and  surrogate  equipment  and  their  impacts  on 
the  test,  identified  by  the  TAWG  must  be  addressed  in  the  Threat 
TSP. 

Appendix  F.  Threat-related  test  limitations.  These  must  be 
reported  to  the  TM  by  the  tester  in  order  for  their  impacts  on  the 
validity  of  the  threat  to  be  portrayed  to  be  assessed  and  described 
in  the  Threat  TSP. 

Appendix  G.  The  threat  force  training  plan.  When  force-on-force 
tests  or  tests  involving  threat  player  personnel  are  required,  the 
supporting  TM  prepares  the  threat  force  training  plan.  It  must  be 
validated  by  TRADOC/CAC  to  ensure  that  the  unit  representing  the 
threat  force  can  execute  the  tactics  and  operations  to  be 
portrayed. 
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Figure  13-6.  Test  profile  sets 
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Factors  influencing  operational  effectiveness. 
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Chapter  14 

Survivability  Considerations  in  Test  and  Evaluation 


Section  I 
Introduction 


14-1.  General 

a.  Survivability  and  mission  performance  are  the  major 
components  of  a  system's  effectiveness.  DOD  policy  requires  that 
the  survivability  of  all  systems  that  must  perform  "critical" 
functions  in  a  man-made  hostile  environment  shall  be  essential 
consideration  durinq  the  acquisition  life  cycle  of  all  proqrams, 
to  include  development  and  non-developmental  programs.  System 
survivability  against  the  full  spectrum  of  battlefield  threats 
found  in  the  various  levels  of  conflict  shall  be  considered  in 
all  systems  acquisition  programs.  These  threats  include 
ballistics;  electronics  warfare;  nuclear  weapon  effects;  nuclear, 
biological  and  chemical  contamination;  directed  energy  as  well  as 
advanced  threats  such  as  high  power  microwave/RF  weapons,  and 
kinetic  energy  weapons.  This  chapter  deals  with  both 
conventional  and  nuclear  survivability  and  describes  the 
survivability  characteristics  and  vulnerability  of  entire 
developing  weapon  systems  (including  crews)  through  T&E. 

b.  Although  the  exact  procedures  used  to  assess  the 
survivability  of  any  system  may  vary,  the  general  approach  is 
similar.  It  must  address  the  relationships  between  the 
avoidance/evasive  and  vulnerability  capabilities  of  the 
system, e.g.,  avoidance  of  being  engaged,  avoidance  of  being  hit 
when  engaged,  and  withstanding  the  effects  when  hit 
(vulnerability) (figure  14-1) . 

(1)  Avoidance  includes  the  tactics,  techniques,  and 
devices  used  to  prevent  detection;  to  evade,  escape,  or  elude 
engagement;  and  avert  being  degraded  by  a  threat.  Avoidance 
includes  dispersion;  deception;  camouflage-cover  concealment;  and 
the  physical,  electronic,  and  audio  non-detection  measures  used 
to  reduce  or  hide  signatures. 

(2)  Vulnerability.  There  is  no  universal  quantitative 
measure  of  survivability — especially  at  the  aggregate  force 
level,  where  survivability  is  complicated  by  employment  of 
systems  in  combination. 

(3)  At  the  system  level,  survivability  is  fairly  well 
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differentiated  from  mission  performance  and  can  be  measured  in 
terms  of  the  fractional  number  of  systems  that  are  capable  of 
continuing  to  perform  their  mission  after  confronting  a  specific 
threat  using  certain  tactics,  techniques,  and  devices  under 
defined  conditions  over  a  finite  period  of  time. 

(4)  This  systems  orientation  is  not  intended  to  deny  the 
importance  of  aggregated  (force  level)  considerations  on 
survivability,  such  as  the  impact  of  proliferation  and  the 
ability  of  other  systems  to  neutralize  a  threat.  Instead,  this 
focus  enables  the  operational  tester  to  hold  force  level 
considerations  constant  so  that  a  direct  comparison  between  the 
old  and  new  systems  can  be  made. 

c.  Survivability  in  test  and  evaluation. 

(1)  Conceptually  Survivability  is  the  product  of  multiple 
series  of  conditional  events  that  start  with  avoiding  detection 
and  extend  through  to  the  ability  to  repair  systems  damaged  or 
disrupted  by  hostile  forces. 

(2)  Estimates  of  survivability  must  take  into  account 
multiple  exposures  to  a  variety  of  threat  systems  in  a 
scenario— dependent  sequence.  Because  such  exposures  to  threat 
systems  are  not  normally  independent,  combat  simulations  often 
are  used  with  variations  in  scenarios  to  estimate  expected  losses 
in  a  combat  environment. 

(3)  A  combat  simulation  may  require  various  estimates, 
such  as — 

(a)  Mobility. 

(b)  Probability  of  detection. 

(c)  Vulnerability  of  the  evaluated  system  to  various 
threat  munitions,  signatures,  and  associated  enemy  means  of 
detection. 

(d)  Projected  threat  doctrine  and  tactics. 

(4)  These  estimates  must  be  determined  by  test  results 
before  a  combat  simulation  is  used.  These  estimates  must  be 
examined  under  varying  factors  and  conditions,  such  as — 

(a)  Day  versus  night  missions. 

(b)  Use  of  countermeasures  and  counter-countermeasures. 
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(c)  Attack  versus  defense  or  other  combat  posture. 

(5)  If  appropriate  models  are  not  available,  one 
alternative  may  be  to  estimate  survivability  by  calculating 
combinations  of  probabilities  of  relevant  factors.  These 
techniques  for  estimating  survivability  most  closely  reflect  the 
approach  used  for  survivability  of  conventional  systems  in  a  non¬ 
nuclear  environment.  They  can  be  adapted,  however,  to  include 
the  effects  of  hostile  EW  against  friendly  communications  and 
computer  systems  as  well  as  to  assess  the  survivability  of  all 
types  of  systems  in  a  nuclear  environment. 

(6)  In  effect,  survivability  of  a  system  in  a  hostile  NBC 
environment  may  be  oriented  toward  the  ability  to  continue 
operations  using  protective  equipment  in  the  nuclear  or  chemical 
environment,  rather  than  surviving  a  "hit”  by  a  nuclear  or 
chemical  munitions.  In  the  case  of  nuclear  and  chemical  warfare, 
suppression  may  occur  only  on  a  strategic  level  and  take  the  form 
of  a  capability  for  massive  retaliation. 

(7)  Thus,  as  a  practical  matter,  OT  conducted  to  help 
resolve  survivability  issues  for  systems  confronting  nuclear  and 
chemical  threats  would  emphasize  the  defensive  countermeasures 
taken  to  protect  the  crew  and  equipment  rather  than  the  strategic 
offensive  actions  to  prevent  first  use  of  such  weapons  or  to  deny 
repeated  use. 

(8)  It  is  important  to  note  that  all  of  these  factors 
assume  knowledge  of  current  and  projected  threat  systems, 
doctrine  and  tactics.  Data  gaps  must  be  identified  immediately 
through  intelligence  channels  to  the  DCSINT  DA  TISO. 

d.  Probability  of  being  engaged. 

(1)  The  probability  of  being  engaged  involves  the 
probabilities  of  being  detected,  acquired,  located,  and  tracked. 
In  some  instances,  an  important  factor  may  be  the  probability  of 
the  evaluated  system  being  identified,  given  its  has  detection. 

(2)  The  probability  of  being  engaged  also  involves  the 
probability  of  suppressing  the  threat  systems  by  friendly  forces. 
Figure  14-2  illustrates  the  factors  that  may  affect  the 
probability  of  being  engaged  and  shows  sample  Drs  for  evaluating 
the  potential  for  engagement. 

(3)  Obtain  as  much  information  as  possible  from  the 
force-on-force  aspects  of  OT,  because  OT  may  provide  estimates  of 
the  true  operational  probability  of  being  engaged. 
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(4)  A  system  may  be  designed  to  reduce  the  probability  of 
engagement  (e.g.,  low  radar  cross-sections  resulting  from  design 
and  construction  with  radar-absorbing  materials).  The  effect  of 
design  characteristics  may  be  negated  or  reduced,  however,  by 
tactics  and  techniques  such  as  night  operations  and  terrain 
following,  and  by  devices  such  as  obscurant  and  jammers. 

e.  Probability  of  being  hit  when  engaged. 

(1)  The  probability  of  being  hit  when  engaged  involves 
such  factors  as  evasive  actions,  suppression,  use  of  decoys  or 
other  techniques,  or  devices  that  prevent  the  evaluated  system 
from  being  hit  by  a  threat  munitions  or  device.  Figure  14-3 
illustrates  example  considerations  for  estimating  the  probability 
of  being  hit  when  engaged. 

(2)  The  selected  methodology  should — 

(a)  Establish  a  baseline  of  threat  effects  against  the 
evaluated  system  under  benign  or  neutral  conditions. 

(b)  Degrade  the  baseline  threat  performance  with 
defensive  countermeasures  (tactics,  techniques,  and  devices) . 

(c)  Further  degrade  the  threat  effectiveness  by 
suppressive  actions.  Suppressing  the  threat  in  this  analysis  does 
not  presume  that  the  enemy  has  fired  (or  acted)  first,  although 
suppressing  an  enemy  who  acts  first  is  decidedly  more  difficult 
than  suppressing  a  less  active  opponent. 

p)  Although  operational  tests  may  provide  information 
regarding  the  probability  of  being  engaged  (and  its  component 
elements) ,  OT  does  not  usually  provide  good  information  on  the 
probability  of  being  hit  or  killed.  Vulnerability  analysis,  live 
fire  survivability  tests,  and  other  analyses  of  threat  impacts 
typically  prove  to  be  better  sources  of  this  information.  It  is 
important,  however,  to  collect  information  regarding  the  relative 
distribution  of  ranges  of  engagement,  typical  angles  of  fire, 
etc.,  that  occur  under  operational  conditions  to  verify  or  modify 
these  inputs  to  the  vulnerability  analysis. 

f.  Probability  of  being  killed  when  hit. 

(1)  Figure  14-4  illustrates  an  example  of  considerations 
for  estimating  the  probability  of  the  system  being  killed  when 
hit  by  a  specific  munitions  or  device. 

(2)  The  ability  to  withstand  threat  effects  when  hit 
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involves  such  factors  as  — 

(a)  Whether  threat  munitions  will  overcome  blue  system 
protection  mechanisms. 

(b)  Whether  flammable  or  explosive  materials  will  be 

hit. 

(c)  Whether  passengers  (in  addition  to  crew  members) 
are  aboard. 

(3)  Because  this  probability  assumes  a  hit,  emphasis  is  on 
the  vulnerability  of  the  system  and  the  designed  hardness, 
robustness,  and  redundancy  of  the  system,  rather  than  on  the 
actions  that  could  be  taken  to  avoid  a  hit-  This  vulnerability 
analysis  of  the  system  is  the  focus  of  TT.  Testing  a  system's 
vulnerability  is  partly  accomplished  by  actually  firing  threat 
munitions  at  the  U.S.  system  (explained  in  the  "Live  Fire  Test 
and  Evaluation  Guidelines"  published  in  support  of  DODM  5000. 2M 
and  in  response  to  the  National  Defense  Authorization  Act  for  FY 
1987  (PL  99-661) ) . 

g.  Ability  to  repair  or  replace  damaged  or  destroyed  systems. 


(1)  Another  aspect  of  survivability  at  the  force  level  is 
the  ability  to  repair  damaged  systems  and  replace  destroyed  or 
lost  systems  (see  fig  14-1).  This  function  overlaps  extensively 
into  the  suitability  area.  Supply  policies,  logistic  system 
capabilities,  and  individual  weapon  system  RAM  all  influence  the 
ability  to  repair  and  replace  systems. 

(2)  Live  fire  testing  may  be  used  to  provide  data  on  the 
operational  requirements  to  repair  combat  damage,  which  often 
cannot  be  determined  during  OT.  You  also  may  consider  the 
benefit  of  allowing  typical  maintenance  personnel  to  participate 
in  the  repair  of  vehicles  subjected  to  live  fire.  Time  required 
for  diagnosis  and  repair  of  battlefield  combat  damage  can  be 
assessed  in  this  manner. 

(3)  Systems  subjected  to  live  fire  typically  are  repaired 
as  quickly  as  possible  for  the  next  phase  of  live  fire  test.  If 
maintenance  by  soldiers  can  be  accommodated,  however,  the^ 
independent  operational  evaluator  gains  valuable  information  on 
adequacy  of  training,  manuals,  tools,  and  test  equipment. 
Although  the  ability  to  repair  and  replace  systems  has 
survivability  implications,  such  logistics  actions  are  more 
properly  addressed  as  suitability  issues.  See  chapter  11  for  a 
discussion  of  RAM  issues,  and  chapter  10  for  logistics 
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supportability. 

h.  Sequence  for  collecting  data  to  support  survivability 
evaluation.  In  the  examination  of  survivability,  the  ability  to 
withstand  threat  effects  when  hit  would  occur  last  in  a  sequence 
beginning  with  the  ability  to  avoid  being  engaged.  In  the  MAP, 
however,  the  survivability  issues  probably  will  be  addressed. 

14-2.  Survivability  measures 

a.  General.  Chapter  4  explains  the  dendritic  process  for 
breaking  down  evaluation  issues  into  MOEs  and  MOPs,  and  MOPs  into 
Drs.  The  preceding  paragraph  illustrated  an  event  dendritic  for 
survivability.  This  paragraph  refines  that  dendritic  by  providing 
factors  and  conditions  that  can  assist  the  independent 
operational  evaluator  in  structuring  OT&E  planning  for 
survivability. 

b.  Survivability  factors  and  conditions.  Two  general  types  of 
factors  determine  system  survivability  of  a  system  and  are 
refined  further  to  satisfy  the  I&C  established  in  the  TEMP.  They 
are,  respectively,  technical  factors,  which  relate  to 
characteristics  of  equipment  such  as  hardness  levels,  system  and 
component  redundancy,  and  robustness,  and  operational  factors, 
which  relate  to  the  tactics,  techniques,  and  devices  used  to 
avoid  destruction,  damage,  or  disruption.  Conditions  are  less 
controllable  variables,  such  as  crew  motivation,  that  are 
influenced  by  determinants  largely  external  to  the  test. 

Technical  factors  and  conditions  are  generally  straight-forward, 
e.g.,  technical  specifications  and  controlled  environments  (see 
Part  Six,  Live  Fire  Testing) .  Operational  factors  and  conditions 
include  such  variables  as  standoff  distance,  dwell  and  exposure 
times,  aspect  angles,  target  area  presented  to  threat  weapons, 
threat  reaction  time  (detection),  and  target  servicing  time. 
Relating  these  factors  to  a  function  or  event  dendritic  produces 
a  framework  for  measurable  operational  survivability  testing.  An 
example  of  a  factors  and  conditions  chart  for  engagement 
avoidance  factors  is  shown  in  table  14-2. 

c.  Evaluation  phases. 

(1)  Operational  evaluation  of  system  survivability  is  done 
in  four  distinct  phases  (see  fig  14-5) : 

(a)  Reviewing  the  survivability  factors,  issues, 
requirements,  and  criteria  developed  in  the  survivability 
analysis. 
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(b)  Assembling  the  results  of  modeling  and  testing 
conducted  to  respond  to  perceived  deficiencies  in  the 
survivability  factors, 

(c)  By  using  planned  tactics,  doctrine,  and 
organization,  determining  whether  the  system  satisfactorily 
overcomes  or  sufficiently  mitigates  those  perceived  survivability 
issues  while  in  an  operational  test  environment  and  thus  meets 
the  survivability  criteria. 

(d)  Determining  the  impact  of  remaining  deficiencies  on 
survivability . 

d.  Review  of  threat  support  package.  Review  the  TSPs  (see  chap 
13)  to  ensure  that  the  threat  force  sufficiently  challenges  the 
system  under  development  and  that  the  survivability  issues  will 
be  answered.  A  useful  aid  in  this  process  is  to  cross-check  the 
factors  and  conditions  matrices  (see  table  14-2)  against  the  TSP. 

e.  Analysis  of  deficiencies.  Analyze  the  parts  of  the  system 
that  are  unable  to  meet  the  established  criteria,  and  determine 
the  degree  to  which  these  deficiencies  adversely  affect 
survivability . 


Nuclear  (residual).  Biological,  Chemical  (NBC)  T  &  E  guidelines 
14-3.  The  need  for  NBC  T&E 

a.  The  objective  of  NBC  T&E  is  to  support  a  timely  and 
thorough  assessment  of  the  survivability  of  a  system  to  the  NBC 
threat  as  it  progresses  through  its  development  an  subsequent 
production  phases.  NBC  contaminants  pose  a  threat  because  they 
can  adversely  affect  mission  performance.  Before  discussing  the 
effects  of  NBC  contamination  upon  mission  performance,  it  must  be 
understood  that  the  ''N”  in  NBC  includes  only  the  secondary 
effects  of  nuclear  detonation  such  as  radioactive  particulate 
fallout  and  the  resulting  induced  gamma  activity.  Primary 
rjuclear  effects  such  as  blast,  initial  radiation,  thermal 
radiation,  and  electromagnetic  pulse  are  covered  in  section  III. 

b.  The  U.S  Army  has  mandated  that  all  equipment  required  to 
perform  mission-essential  functions  must  be  survivable  in  a 
nuclear^  biological,  and  chemical  (NBC)  environment.  This 
section  describes  the  key  steps  for  developing  an  acceptable  NBC 
test  strategy  including  the  role  of  modeling  and  testing.  It 
also  providss  guidancs  on  the  planning,  execution,  repoirting,  and 
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review  and  approval  process  and  outlines  the  roles  of  key 
activities  essential  to  the  process. 

14-4.  Primary  NBC  Threat 

a.  The  primary  threat  of  NBC  contamination  to  mission 
performance  is  the  adverse  effects  these  contaminants  have  upon 
personnel.  In  addition,  the  performance  of  a  mission  may  be 
degraded  because  NBC  contamination  cause  direct  degradation  of 
military  equipment  or  indirectly  hinders  its  employment.  NBC 
contamination  can  have  direct  effects  upon  equipment  by  degrading 
critical  properties  (e.g.,  physical,  chemical,  mechanical, 
electrical,  optical,  or  thermal)  of  the  materials  of 
construction.  In  addition,  materials  or  procedures  used  to 
decontaminate  equipment  items  also  may  degrade  the  critical 
properties  of  materials.  The  degradation  of  these  properties 
will,  in  turn,  reduce  the  capability  of  a  component  or  system  to 
perform  an  intended  function.  Indirect  effects  may  arise  if  a 
system  cannot  be  decontaminated  to  levels  which  are  not  hazardous 
to  unprotected  personnel.  As  a  result,  operators  will  be  forced 
to  wear  individual  protective  clothing,  which  inhibits  their 
performance.  Limitations  caused  by  the  wearing  of  protective 
clothing  include  loss  of  tactile  capabilities,  reduced  vision, 
and  reduced  work  capacity.  Decontamination  of  a  surface  may  be 
hindered  because  the  material  of  construction  absorbs  the 
contaminant,  or  the  contaminant  is  trapped  in  cracks  or  crevices. 

b.  The  threat  posed  by  NBC  contamination  has  been  heightened 
due  to  the  proliferation  of  NBC  weapons  by  NATO,  Third  World,  and 
former  Soviet/Warsaw  Pact  nations.  These  weapons  of  mass 
destruction  are  becoming  more  readily  available,  technological 
advances  are  continuing  to  increase  their  military  effectiveness 
and  worth.  Also,  it  has  been  estimated  that  14  to  16  nations 
posses  stocks  of  chemical  weapons  and  the  capability  to  conduct 
chemical  warfare  (CW) . 

14-5.  Biological  Warfare  (BW) 

Biological  warfare  capabilities  are  more  difficult  to  gather  and 
verify  because  the  production  and  dissemination  of  biological 
agents  can  easily  be  disguised.  Facilities  normally  used  to 
manufacture  pharmaceutical  can  be  operated  to  produce  BW  agents 
and  toxins.  Dissemination  of  BW  agents  can  be  disguised  as 
natural  occurrences.  In  addition,  the  threat  of  BW  agents  and 
toxins  has  been  enhanced  because  of  the  evolution  of  genetic 
engineering.  This  technology  provides  a  means  of  expanding  the 
number  of  toxic  BW  agents  and  toxins,  increase  their  lethality, 
and  defeat  the  effectiveness  of  existing  antidotes  and  vaccines. 
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14-6.  NBC  Survivability  Requirements 

a.  The  requirement  for  NBC  contamination  survivable  equipment 
resulted  from  growing  capability  of  potential  US  adversaries  to 
employ  NBC  weapons  and  the  possibility  that  contamination  of 
material  by  these  weapons  may  degrade  mission  performance.  In 
response  to  the  threat,  the  US  Army  (lead  agency  for  providing  CB 
defense  capability)  issued  Army  Regulation  (AR)  70-71,  "Nuclear, 
Biological,  and  Chemical  Survivability  of  Army  Material,"  dated  1 
April  1984.  This  regulation  states  that  all  mission-essential 
items  or  critical  components  of  one  or  more  mission-essential  end 
items  must  be  NBC  contamination  survivable.  Mission  essential 
material  is  defined  to  be  that  necessary  to  accomplish  the 
primary  or  secondary  function  of  a  military  unit  or  organization. 
The  nuclear  hazard  is  limited  to  the  secondary  nuclear  effects  or 
i^esidual  radiological  contamination  consisting  of  fallout, 
rainout,  and  neutron  induced  gamma  activity.  These  residual 
hazards  are  distinguished  from  primary  nuclear  effects  of  blast, 
thermal,  initial  radiation,  and  electromagnetic  pulse  (EMP)  which 
are  addressed  in  Ar  70-60. 

b.  In  a  manner  similar  to  its  nuclear  counterpart  (AR  70-60), 
the  NBC  contamination  survivability  regulation  (AR  70-71) 
establishes  the  policy  and  procedures  for  developing  and 
acquiring  NBC  survivable  material.  This  regulation  states  that 
NBC  contamination  survivability: 

(1)  Will  be  considered  in  all  material  requirements 
documents . 

(2)  Will  have  its  criteria  established  during  the  program 
initiation  phase  and  refined.  If  necessary,  these  criteria  will 
be  established  during  advanced  development. 

(3)  Will  be  managed  throughout  the  life  cycle. 

(4)  Will  have  any  criteria  changes  or  waivers  adjudicated 
only  by  the  Nuclear  and  Chemical  Survivability  Committee. 

14-7.  Regulatory  Guidance  and  Responsibilities 

a.  The  regulation  also  identifies  the  various 
responsibilities  of  HQDA,  MACOMs  and  other  agencies  in  the 
development  and  implementation  of  the  NBC  contamination 
survivability  programs.  Of  particular  interest  to  the  equipment 
developers  are  the  responsibilities  of  the  US  Army  Training  and 
Doctrine  Command  (TRADOC) ,  the  US  Army  Nuclear  and  Chemical 
Agency  (USANCA) ,  and  the  Army  Material  Command  (AMC) . 
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(1)  TRADOC  is  responsible  for  ensuring  that  NBC 
contamination  survivability  of  material  is  considered  early  in 
the  development  cycle  and  that  the  survivability  criteria  are 
included  in  requirements  documents.  TRADOC  policy  and  procedures 
are  outlined  in  TRADOC  Regulation  71-14. 

(2)  USANCA  is  responsible  for  providing  the  NBC 
contamination  survivability  criteria. 

(3)  AMC  is  responsible  for; 

(a)  Establishing  and  maintaining  a  technology  base  in 
support  of  survivability. 

(b)  Developing  Independent  Evaluation  Plans  (lEPs)  and 
reports  (lERs)  to  assess  survivability. 

(c)  Providing  technical  advice  and  assistance  to  all 
material  developers. 

(d)  Serving  on  various  committees  where  NBC 
survivability  issues  are  of  concern. 

b.  AMC  has  designated  the  Chemical  Research,  Development,  and 
Engineering  Center  (CRDEC)  the  lead  laboratory  to  set  up  a 
technical  assistance  program  to  support  all  material  developers. 
CRDEC  assists  and  supports  the  preparation  of  Requests  for 
Proposals  (RFPs) ,  Request  for  Quotations  (RFQs)  and  Statements  of 
Work  (SOWS) .  CRDEC  also  participates  in  Source  Selection 
Evaluation  Boards  (SSEBs)  and  at  key  Design  Reviews.  In 
addition,  CRDEC  is  assigned  the  responsibility  to  brief  all  Major 
Subordinate  Commands  on  survivability  and  serve  as  a  member  of 
the  Nuclear  and  Chemical  Survivability  Secretariat.  Guidance  for 
contractors  and  material  developers  is  provided  in  a  CRDEC 
publication,  NBC  Contamination  Survivability:  A  Handbook  for 
Development/Management  of  Material  Programs. 


Section  III 

Electromagnetic  Environmental  Effects  (E3) 


14-8.  General 

a.  This  Section  of  the  Test  and  Evaluation  (T&E)  Procedures 
Guide  is  intended  to  provide  guidance  to  acquisition  managers  on 
the  integration  of  the  Electromagnetic  Environmental  Effects  (E3) 
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program  into  T&E  processes.  Electromagnetic  compatibility  and 
vulnerability  testing  and  analysis  will  be  conducted  as  part  of 
the  system's  normal  life-cycle  T&E  program  in  accordance  with 
Army  E3  test  procedures. 

b.  The  objectives  of  the  Army's  E3  program  are  to  prevent 
electromagnetic  interference,  maintain  electromagnetic 
compatibility  standards  and  specifications,  achieve 
electromagnetic  compatibility  for  all  electrical  or  electronics- 
dependent  equipment,  and  attain  built-in  design  compatibility 
through  the  use  of  after-the-fact  remedies.  However,  it  is  not 
always  feasible  to  protect  every  systems  or  subsystem  from  E3 . 

If  the  system  performance/raeasurements  are  less  than  the 
established  E3  criteria,  the  level  of  compliance  must  be 
determined  to  access  the  impact  on  safety  and  miscion  or 
operational  effectiveness  throughout  the  life  cycle. 

c.  Defense  acquisition  policy  states  that  new  systems  must 
neither  suffer  nor  cause  unacceptable  mission  degradation  due  to 
electromagnetic  environmental  effects.  The  Army  E3  Program 
implements  Defense  policy  by  insuring  that  all  acquisitions 
follow  a  consistent  set  of  processes:  criteria  development, 
assessment,  planning  and  execution.  An  E3  Requirements  Board 
(E3RB) ,  composed  of  the  Materiel  Developer  (MATDEV) ,  the  Combat 
Developer  (CBTDEV) ,  and  matrix  support  engineer  from  the  Army 
Materiel  Command's  (AMC)  Major  Subordinate  Command  (MSC) ,  is 
formed  for  each  program  to  direct  and  oversee  the  processes,  and 
make  recommendations  to  the  program  decision  authority. 

(1)  Implicit  in  the  E3  program  is  the  understanding  that 
it  is  neither  feasible  nor  affordable  to  achieve  perfect 
protection.  The  Army  E3  Program  is  restricted  to  elimination  of 
unacceptable  E3  from  materiel  acquired  for  military  use.  Thus  it 
becomes  important  to  determine  effects,  assess  their  severity  and 
likelihood  of  occurrence,  and  balance  risk  against  mission 
success,  while  maintaining  high  standards  of  safety.  The  E3RB 
plays  the  key  role  in  assessing  the  impact  of  effects,  and 
whether  a  hardware  or  operational  'fix'  is  appropriate.  The  E3RB 
is  the  primary  E3  advisor  to  the  program  decision  authority,  and 
works  closely  with  the  T&E  community  through  the  Test  Integration 
Working  Group  (TIWG)  to  assure  a  smooth  integration  of  test 
results,  particularly  when  there  are  failures. 

(2)  The  Army  E3  Board  is  responsible  for  assuring  the 
continuity  and  consistency  of  the  Army  E3  Program  by  bringing 
together  representatives  of  AMC  MSCs  and  other  materiel 
developers,  the  Training  and  Doctrine  Command  (TRADOC) ,  and 
Department  of  the  Army  elements  in  a  common  forum.  The  E3  Board 
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has  a  permanent  working  group  concerned  with  T&E  policy  and 
process,  but  does  not  get  involved  with  individual  acquisition 
programs . 

d.  In  the  broadest  sense,  E3  includes  any  effect  caused  by 
electromagnetic  energy,  regardless  of  source,  intent,  or 
consequence.  Friendly,  and  hostile  emitters  are  included  as  well 
as  naturally  occurring  phenomena.  However,  the  principle  focus 
of  the  Army  E3  program  is  on  the  unintentional  or  accidental 
effects  of  tHe  electromagnetic  environment  on  military  systems. 
The  E3  program  encompasses  all  electromagnetic  disciplines, 
including  electromagnetic  compatibility  (ECM) ,  electromagnetic 
interference  (EMI) ,  electrostatic  discharge  (ESD) ,  hazards  of 
electromagnetic  radiation  to  personnel,  ordnance  and 
fuels/volatile  materials  nuclear  electromagnetic  pulse  (EMP) . 

(1)  Electromagnetic  radiation  is  an  oscillating  electric 
and  magnetic  field  propagated  through  space  at  the  speed  of 
light.  The  E3  program  is  concerned  with  that  portion  of  the 
electromagnetic  spectrum  from  the  lowest  alternating  current  (AC) 
frequency  (approximately  1  Hz)  to  the  highest  microwave  frequency 
(approximately  300gHz) .  Electromagnetic  radiation  of  higher 
frequency  (i.e.,  shorter  wavelength),  classified  as  infrared, 
visible  (i.e.,  light),  ultraviolet.  X-rays,  and  gamma  rays  is  not 
part  of  the  Army  E3  program. 

(2)  Sources  of  electromagnetic  radiation  may  be  from 
intention  emitters  such  as  radio/TV  transmitters  and  radar, 
unintentional  emitters  such  as  power  lines  and  electrical  motors, 
or  natural  phenomena  such  as  lightning  and  static  discharge. 
Electromagnetic  energy  may  be  of  a  continuous  wave  (CW)  nature  or 
it  may  be  of  a  transient  or  pulsed  nature.  CW  energy  is 
described  by  its  frequency  and  power  level  or  amplitude  while 
transient  electromagnetic  energy  is  measured  in  time  and 
amplitude  terms  such  as  duration,  risetime  (from  start  to  peak), 
and  peak  amplitude.  Examples  of  CW  sources  include  radio 
transmitters  (a  radiating  source)  and  power  lines  (a  conducting 
source) .  Examples  of  transients  are  lightning,  electrostatic 
discharge  (ESD) ,  and  a  nuclear  weapon  generated  electromagnetic 
pulse  (EMP) .  Electrical  noise  generated  by  electrical  switching 
and  motors  falls  in  the  category  of  transients. 

(3)  Electromagnetic  effects  may  range  from  audio  or  video 
interference/noise  or  disruption,  control  system  malfunction,  or 
erroneous  or  imprecise  sensing.  Effects  may  range  in  severity 
from  a  barely  detectable  threshold  level,  to  catastrophic  damage. 
The  absorption  of  electromagnetic  energy  in  munitions  and  fuels 
may  be  sufficient  to  cause  unintentional  firing  or  detonation. 


14-12 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992 


(DRAFT) 


(4)  A  well  planned  E3  testing  program  must  be  conducted  at 
two  levels.  A  sub-system  test  is  performed  on  a  functional  part 
of  the  larger  system,  and  will  normally  require  a  simulated  local 
environment,  connections  to  input  and  output  ports  of  the  item 
under  test,  and  where  appropriate,  simulated  signal  traffic. 
Sub-systems  are  tested  to  evaluate  sensitivity  to  the  local 
(intra-system)  electromagnetic  environment,  including  that  from 
other  sub-systems. 

(5)  System  level  testing  is  normally  accomplished  by 
illuminating  the  test  system  with  a  uniform  field,  or  a 
reasonable  spot  illumination,  applied  piecemeal  over  the  system. 
Spectrum  restrictions  may  preclude  a  full  set  of  realistic 
frequencies  in  open  air  testing,  thus  limiting  inter-system 
effects  from  other  radiating  systems.  Laboratory  tests,  in 
shielded  enclosures,  anechoic  chambers,  or  other  special 
facilities  must  be  used  when  open  air  testing  is  not  feasible. 

New  techniques  of  evaluating  full  systems  are  transverse 
electromagnetic  wave,  reverberation  (TEM/REV)  chambers,  and 
synchronous  current  injection  of  all  system  coupling  points. 

14-9.  Functional  Relationships 

a.  The  functional  roles  of  the  major  Head  Quarters  Department 
of  the  Army  (HQDA)  staff  and  Army  components  relative  to  the 
implementation  of  the  Army  E3  Program  are  provided  below.  Only 
those  roles  which  are  specific  to  the  Army's  E3  Program  are 
included. 

b.  Headquarters,  Department  of  the  Army  (HQDA) . 

(1)  The  Director  of  Program  and  Vulnerability  Assessment, 
Office  of  the  Assistant  Secretary  of  the  Army  (Research, 
Development  and  Acquisition)  (SARD-D) : 

(a)  Serves  as  the  Army  focal  point  for  E3  policy  and 
program  implementation; 

(b)  Develops  Army  E3  policy  guidance; 

(c)  Maintains  oversight  of  E3  policy  implementation; 

(d)  Provides  overall  assessment  of  Program  Executive 
Officer/Program  Manager  (PEO/PM)  E3  program  accomplishments  for 
system  acquisition  milestone  decision  reviews. 

(2)  The  Deputy  Chief  of  Staff  for  Intelligence; 
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(a)  Develops  and  maintains  non-United  States  (US) 

(i.e.,  enemy,  allied  and  friendly)  electromagnetic  emitter 
information  and  databases. 

(3)  The  Spectrum  Manager: 

(a)  Establishes  policy  on  the  utilization  of  the 
electromagnetic  spectrum  for  Army  Systems; 

(b)  Approves  system  operating  frequency  requests  (SF 
Form  1494) . 

(4)  The  PEO/PM: 

(a)  Incorporates  Army  E3  policy  into  system  development 
and  acquisition  programs; 

(b)  Provides  a  voting  member  to  the  E3RB; 

(c)  Utilizes  the  approved  E3  Criteria  as  baseline 
system  performance  requirements  and  incorporates  the  E3  Criteria 
into  the  system  specification; 

(d)  Assures  that  adequate  testing  is  planned  and  funded 
to  determine  system  performance  in  the  established  E3 
environment . 

(5)  The  E3  Requirements  Board: 

(a)  Is  established  for  each  system  and  is  composed  of 
one  voting  member  from  the  MATDEV  (i.e.,  PEO/PM  or  AMC  MSC) , 
CBTDEV  (i.e.,  TRADOC) ,  and  the  AMC  MSC  which  provides  engineering 
support  to  the  MATDEV; 

(b)  Develops  and  approves  system  specific  E3  operating 
environments ; 

(c)  Establishes  quantitative  E3  Criteria  based  upon  the 
Operational  Requirements  Document  (ORD)  and  the  approved  E3 
operating  environment; 

(d)  Provides  recommended  E3  Criteria  to  the  PEO/PM; 

(e)  Reviews  adequacy  of  system  performance  against  the 
approved  E3  criteria; 

(f)  Recommends  technical/operational  solutions  or 
modifications  of  the  approved  E3  Criteria. 
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(6)  The  Army  E3  Board: 

(a)  Composed  of  representatives  from  HQ  DA,  Army  Staff, 
HQ  AMC,  AMC  MSCs  and  Research  Development  and  Engineering  Centers 
(RDECs) ,  Army  Materiel  Systems  Analysis  Activity  (AMSAA) ,  TRADOC, 
US  Army  Information  Systems  Command  (USAISC) ,  Army  Intelligence 
Agency  (AIA) ,  and  Strategic  Defense  Command  (SDC) ; 

(b)  Provides  coordination  of  E3  Program  activities 
between  participating  organizations; 

(c)  Acts  as  a  clearinghouse  for  E3  information  and 
serves  as  the  focal  point  for  standardization  for  E3  criteria 
and  assessment  methodology; 

(d)  Manages  the  formulations  and  modifications  to 
Mission  Area  E3  environments; 

(e)  Provides  technical  advice  and  support  to  HQ  DA  and 
the  Army  Staff. 

c.  Army  Materiel  Command  (AMC) . 

(1)  The  Deputy  Chief  of  Staff  for  Acquisition  (AMCAQ-AP) : 

(a)  Serves  as  the  AMC  focal  point  for  E3  policy  and 
program  implementation; 

(b)  Provides  E3  oversight  and  coordinates  support  of 
AMC  activities. 

(2)  The  Army  Materiel  System  Analysis  Activity  (AMSAA): 

(a)  Reviews  Operational  Requirements  Document  (ORD)  to 
determine  if  the  mission  statement  and  operational  concept  are 
adequate  to  establish  an  E3  operating  environment; 

(b)  Identifies  E3  critical  issues  and  incorporates  E3 
considerations  into  the  Independent  Evaluation  Plan  (lEP) ; 

(c)  Prepares  Test  Design  Plans  (TDPs)  and  incorporates 
adequate /appropriate  data  requirements  necessary  to  evaluate  E3 
critical  issues; 

(d)  Participates  through  the  Test  Integration  Working 
Group  (TIWG)  process  in  planning  developmental  and  operational 
testing; 
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(e)  Evaluates  E3  performance  against  E3  criteria; 
documents  E3  test  results  and  evaluations  in  the  Independent 
Evaluation  Report  (lER) ; 

(f)  Provides  a  member  to  the  Army  E3  Board. 

(3)  The  Test  and  Evaluation  Command  (TECOM) : 

(a)  Identifies  E3  critical  issues  and  incorporates  E3 
criteria  in  the  Independent  Assessment  Plan  (lAP) ; 

(b)  Prepares  Test  Design  Plans  (TDPs)  and  incorporates 
adequate/appropriate  data  requirements  necessary  to  evaluate  E3 
critical  issues; 

(c)  Participates  through  the  TIWG  process  in  planning 
developmental  testing; 

(d)  Evaluates  E3  performance  against  E3  criteria; 
documents  E3  test  results  and  evaluations  in  the  Independent 
Assessment  Report  (lAR) ; 

(f)  Develops  and  maintains  comprehensive  E3  test 
facilities  and  supporting  capabilities; 

(g)  Conducts  E3  testing  in  support  of  system 
development  and  testing  programs; 

(h)  Develops  and  maintains  US  (i.e.,  military  and 
civilian)  electromagnetic  emitter  information  and  databases. 

(i)  For  those  systems  assessed  by  TECOM,  reviews  ORDs 
to  determine  if  the  mission  statement  and  operational  concept  are 
adequate  to  establish  an  E3  operating  environment. 

d.  Training  and  Doctrine  Command  (TRADOC) . 

(1)  TRADOC: 

(a)  Incorporates  E3  considerations  into  operational 

concepts ; 

(b)  Establishes  qualitative  E3  requirements  for 
incorporation  into  requirements  documents  (i.e.,  ORD) ; 

(c)  Participates  as  a  voting  member  of  the  E3 
Requirements  Board. 
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(d)  Provides  a  member  to  the  Army  E3  Board. 

e.  Operational  Test  and  Evaluation  Command  (OPTEC) . 

(1)  OPTEC: 

(a)  Manages  Continuous  Evaluation  (CE)  and  Operational 
Testing  Programs; 

(b)  Manages  the  User  Testing  Program  and  evaluates 
testing  processes  to  continuously  improve  these  processes  for  the 
purpose  of  optimizing  resources  and  improving  products. 

(2)  The  Operational  Evaluation  Command  (OEC) : 

(a)  Reviews  the  system  COIC  to  determine  if  E3  should 
be  included  in  the  COIC; 

(b)  Executes  CE  Programs  to  ensure  timely,  complete, 
and  independent  operational  evaluations. 

(3)  The  Test  and  Experimentation  Command  (TEXCOM) : 

(a)  Conducts  operational  testing  to  support  CE  and 
force  development. 

(b)  Plans  and  conducts  realistic  operational  testing 
which  includes  scenarios  which  expose  the  system  to  the  projected 
electromagnetic  environment. 

f.  MEDICAL  RESEARCH  AND  DEVELOPMENT  COMMAND  (MRDC) . 

(1)  Conducts  research  on  the  effects  of  electromagnetic/rf 
radiation  on  personnel; 

(2)  Recommends  appropriate  safety  standards  for  personnel 
exposed  to  hazardous  electromagnetic  environments. 

14-10.  Test  and  Evaluation  Requirements 

a.  Army  guidance,  as  stated  in  this  PAM,  requires  a  system's 
E3  requirement  be  identified  to  the  Test  and  Evaluation 
Management  Agency  (TEMA) ,  and  that  the  initial  strategy  and 
resource  requirements  be  included  in  the  Milestone  I  TEMP.  The 
TEMP  is  the  basic  planning  document  by  which  the  Army  coordinates 
and  approves  the  E3  test  strategy  for  a  given  system.  The  PM  is 
responsible  for  preparing  the  TEMP,  in  conjunction  with  the  TIWG. 
It  is  prepared  in  accordance  with  the  guidance  contained  in  DOD 
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b.  Test  and  Evaluation  Master  Plan.  The  TEMP  defines  and 
integrates  test  objectives,  critical  issues,  system 
characteristics,  responsibilities,  resources,  and  schedules  for 
T&E.  The  TEMP  summarizes  what,  why,  who,  where,  when,  and  how 
the  E3T&E  issues  will  be  tested  and  evaluated.  All  E3T&E  which 
impacts  on  program  decisions  will  be  outlined  in  the  TEMP.  The 
TEMP  shows  the  relationship  of  the  E3T&E  issues  to  the  required 
developmental  and  operational  characteristics;  describes  the 
critical  vulnerability  issue  and  evaluation  criteria;  outlines 
the  planned  E3T&E;  discusses  the  amount  and  type  of  E3T&E  that 
will  be  performed  to  support  each  program  decision  point;  and 
indicates  where  schedule,  resource  or  budget  constraints  may 
impact  the  adequacy  of  planned  E3T&E.  The  primary  E3T&E  resource 
requirements  should  be  identified  and  addressed  in  the  T&E 
Resource  Section  of  the  TEMP  as  early  as  possible,  to  facilitate 
budget/ schedule  projections;  initial  resource  requirement  should 
be  identified  prior  to  the  Milestone  I  decision.  This  will  also 
insure  the  early  identification  and  programming  of  funds  required 
for  test  execution.  AMSAA/the  Independent  Developmental 
Evaluator  reviews  the  TEMP  and  determines  if  the  proposed  testing 
is  adequate  to  evaluate  the  E3  capabilities/ limitations  of  the 
system.  OPTEC/the  Independent  Operational  Evaluator  also  reviews 

the  TEMP  to  determine  if  the  proposed  testing  is  adequate  to  | 

validate  E3  operational  considerations.  Detailed  guidance  for  ' 

the  preparation  of  the  TEMP  is  contained  in  Part  II  of  this 
Pamphlet. 

c.  Test  Integration  Working  Group.  During  the  Concept 
Exploration/Definition  Phase,  the  PM  establishes  the  TIWG  to 
facilitate  integration  of  the  T&E  requirements.  The  TIWG  serves 
as  the  forum  for  planning,  coordination  and  integration  of  the 
E3T&E  program.  The  TIWG  is  discussed  in  greater  detail  in 
Chapter  8  of  this  Pamphlet.  The  TIWG  is  assisted  in  defining  the 
E3T&E  program  and  in  preparing  the  TEMP  by  the  E3  Requirement 
Board  member,  who  is  appointed  by  the  PM  during  the  Concept 
Exploration/Definition  Phase.  The  TIWG,  with  the  assistance  of 
E3  Requirements  Board,  updates  the  TEMP  in  preparation  for  each 
Major  Milestone  Review. 

d.  E3  Requirements  Board.  The  E3  Requirements  Board 
determines  initial  E3  criteria;  evaluates  the  feasibility  of 
meeting  the  criteria;  conducts  mission  and  hardening  level  trade¬ 
off  analysis;  and  makes  recommendations  to  the  PM  and  the  TIWG. 


14-11.  Developmental  Testing  and  Evaluation 
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a.  This  section  will  address  E3  testing  and  evaluation  as  it 
relates  to  the  Army's  Technical  Test  and  Evaluation  (TT&E) 
requirements.  This  section  will  describe  the  following  as  it 
pertains  to  TT&E  planning,  resources,  execution,  and  reporting. 

(1)  Planning  Independent  Evaluation/Assessment  Plan 
(lEP/IAP) .  The  lEP/IAP  is  prepared  in  accordance  with  the 
procedures  contained  in  Part  IV  of  this  PAM.  Additional 
considerations  are  addressed  below. 

(a)  If  E3  has  been  determined  to  be  a  critical 
technical  parameter  for  a  particular  system,  it  will  be  cited  in 
the  TEMP  well  as  in  the  lEP/IAP.  If  E3  is  not  a  critical 
technical  parame  ter,  but  is  to  be  addressed  during  Developmental 
T&E,  it  can  be  cited  in  the  IEP\IAP  as  a  noncritical  technical 
parameter.  It  is  made  up  of  several  sub-elements.  Sub-elements 
that  may  be  addressed  are  as  follows:  Electromagnetic 
Interference  (EMI) ,  Electromagnetic  Compatibility  (EMC) , 
Electromagnetic  Radiation  Operational  (EMRO) ,  Electromagnetic 
Radiation  Hazard  (EMRH) ,  Electrostatic  Discharge  (ESD) , 
Electromagnetic  Pulse  (EMP) ,  Lightning  Effects  (LE) . 

(b)  The  criteria  for  these  elements  are  defined  in  the 
lEP/IAP  and  are  obtained  from  current  Army  E3  Military  Standard 
(MIL-STD)  requirements,  as  well  as  those  developed  by  the  E3 
Requirements  Board  of  the  system.  Figure  14-7  provides  a  generic 
overview  of  the  E3  critical  technical  parameter  as  it  might 
appear  in  an  lEP/IAP. 

(c)  Test  Design  Plan  (TDP) .  Prior  to  Developmental 
Feasibility  Testing,  a  TDP  shall  be  prepared  in  accordance  with 
Part  IV  of  this  Pamphlet  to  outline  specific  testing  and  data 
requirements.  Additional  considerations  are  listed  below. 

(1)  The  TDP  is  a  guide  to  the  developmental  tester  in  the 
development  of  the  Developmental  tester's  Detailed  Test  Plan 
(DTP) . 


(2)  Figure  14-8  provides  a  flowchart  to  assist  in 
identifying  necessary  E3  testing. 

(3)  Once  the  E3  test  matrix  is  determined,  the  TDP  will 
outline  the  system  test  type,  test  criteria,  and  test  data 
collection  necessary  for  AMSAA's  independent  evaluation. 

Table  14-3  provides  general  references  in  determining  a  system, 
subsystem/ component,  and  ordnance  level  testing,  test  criteria, 
and  test  operating  procedures  for  E3  testing. 

(4)  EMI,  EMC,  ESD,  LE,  EMP,  and  EMRH  have  straightforward. 
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Critical  Issue;  How  well  does  the  system  operate  in  its 
intended  E3  environment? 

Scope ;  This  issue  will  examine  the  effects  of  various 
Electromagnetic  Environments  (EME)  on  the  ability  of  the 
system  to  be  transported,  stored,  and  operated.  This  issue 
is  concern'ed  with  the  effects  of  unintentionally-coupled  EM 
radiation  (as  opposed  to  intentionally  directed  EM)  that  the 
system  is  expected  to  encounter.  The  E3  critical  issue  is 
comprised  of  six  sub-issues: 

Electromagnetic  Interference  (EMI) 

-  Electromagnetic  Radiation  Operational  (EMRO) 
Electromagnetic  Radiation  Hazard  (EMRH) 

-  Electrostatic  Discharge  (ESD) 

-  Electromagnetic  Pulse  (EMP) 

Lightning  Effects  (LE) . 

Criteria:  The  system  will  comply  with  E3-related  design 

standards/requirements,  MIL-STD  requirements  as  they  pertain 
to  the  system,  and  the  operational  electromagnetic  environment 
as  determined  by  the  E3  Requirements  Board  (E3RB) . 

Methodology:  The  system  will  undergo  E3  testing  to 

determine  compliance.  Anomalies  will  be  assessed  for  their 
impact  on  the  system's  operation. 


Figure  14-1.  An  example  of  an  E3  entry  in  the  AMSAA  lEF 


unambiguous  test  requirements,  criteria,  and  test  procedures  as 
outlined  in  the  appropriate  E3  documents.  However,  since  EMRO 
considers  the  operational  EM  environment  of  the  system,  EMRO  is 
by  definition  dependent  upon  the  deployment  of  the  system.  The 
test  criteria  necessary  to  evaluate  the  EMRO  criteria  is  provided 
by  the  E3RB  of  the  system.  AMSAA  adopts  the  EMRO  criteria  in  its 
independent  evaluation.  When  a  draft  TDP  is  compiled  it  is 
coordinated  with  the  TIWG  members.  When  coordination  is 
completed  and  the  TDP  is  approved  by  AMSAA  management,  the  TDP  is 
provided  to  the  developmental  tester  for  use  in  developing  the 
DTP. 


b.  Resourcing.  T&E  resources  should  be  in  accordance  with 
Part  IV  of  this  Pamphlet.  Refer  to  Table  14-4  of  this  Section 
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for  specific  test  center  resources  for  E3,  and  Table  14-11  for  E3 
(Insert  fig. 14-7) 

c.  Execution.  Test  execution  shall  be  in  accordance  with  art 
IV  of  this  Pamphlet.  Part  IV  of  this  Pamphlet  specifically 
addresses  E3 . 

d.  Test  Reporting.  Test  Incident  Reports  (TIRs)  and  test 
reports  (TRs)  should  be  done  in  accordance  with  Part  IV  of  this 
Pamphlet. 

(1)  Independent  Evaluation/Assessment  Report  (lER/IAR) . 

The  purpose  of  the  lER/IAR  is  to  document  the  results  of  the 
independent  evaluation/assessment.  They  should  be  prepared  in 
accordance  with  Part  IV  of  this  Pamphlet.  As  E3  is  one  of  the 
critical  technical  parameters  evaluated  in  the  lER/IAR,  the 
results  from  E3  testing  are  compared  with  user 
requirements  and  contract  specifications  to  determine  the 
system's  compliance  with  E3  test  criteria.  Additional 
considerations  are  addressed  below. 

(a)  The  independent  evaluation/assessment  is  performed 
based  on  results  from  the  system  testing,  modeling/ simulation, 
and  other  data  sources  (e.g.,  E3RB  evaluations,  contractor  test 
reports,  component  test  reports,  etc.). 

(b)  To  complete  the  independent  evaluation/assessment, 
the  results  of  the  testing  and  modeling  are  compared  to  the 
performance  criteria  of  the  system.  When  a  system  does  not  meet 
the  specified  criteria,  then  an  analysis  is  conducted  to 
determine  how  the  operation  of  the  system  is  impacted  by  not 
meeting  the  criteria.  The  results  of  the  E3RB  as  documented  by 
the  minutes  are  used  to  assist  the  evaluators/assessors  in  their 
impact  assessment.  Every  E3  anomaly  found  in  the  system  is 
analyzed  to  determine  how  the  anomaly  degrades  the  mission  of  the 
system. 


(c)  Figure  14-9  provides  an  overview  of  the  E3  test  and 
evaluation  process.  There  are  two  possible  approaches  in 
conducting  E3  anomaly  investigations.  One  approach  identifies 
the  anomaly,  conducts  a  fix  (e.g.,  additional  shielding, 
rebonding,  etc.)  and  retests.  Another  approach  assesses  impact 
of  the  anomaly  on  the  test  item's  operation  or  performance.  It 
is  advisable  to  attempt  to  fix  the  identified  anomaly  whenever 
feasible  such  that  the  item  under  test  will  meet  its  E3  criteria. 
However,  the  cause  of  many  anomalies  may  never  be  adequately 
identified  or  may  be  expensive  to  fix.  In  the  latter  case,  the 


14-21 


DA  Pamphlet  73-1,  Part  One,  16  Oct  1992 


(DRAFT) 


E3RB  will  evaluate  the  anomaly,  assess  its  impact  on  the  test 
item's  operation/performance,  and  provide  a  recommendation  on  the 
future  course  of  action  based  on  the  impact  of  the  anomaly.  The 
anomaly  impact  can  be  one  of  three  possibilities:  anomaly  poses 
little  or  no  risk  in  test  item  operation  or  mission  impact  (this 
will  require  a  waiver) ;  anomaly  poses  risk  in  test  item 

operation  but  an  operational  fix*  will  address  the  risk  (this 
will  require  a  waiver) ;  or  anomaly  poses  risk  in  test  item^ 
operation  and  will  require  a  technical  fix  (this  will  require  a 
retest) . 


(d)  Once  the  impact  is  assessed,  the  evaluator/assessor 
will  determine  the  ability  of  the  system  to  operate  in  its  E3 
environment  and  provide  a  technical  risk  assessment  of  the  system 
in  meeting  its  E3  performance  requirement. 

14-12.  Operational  Testing 

a.  Operational  testing  applies  to  all  materiel  systems 
acquired  under  the  AR  70  series  regulations  and  all  information 
management  area  (IMA)  systems  acquired  under  the  AR  25  series. 

The  different  varieties  of  each  of  these  main  categories  that  are 
discussed  in  Part  V  of  this  Pamphlet.  Operational  tests  are 
designed  to  answer  whether  troops  can  operate,  maintain,  and 
support  the  system  when  it  is  deployed  in  an  operational 
environment.  The  common  denominator  for  the  different  varieties 
of  user  testing  is  participation  of  typical  user  troops  operating 
the  system  under  test  in  an  operational  environment  that  is  close 
to  operationally  realistic.  The  operational  environment  includes 
tactical  operations  conducted  in  accordance  with  wartime 
operational  mission  profiles  as  planned  for  tactics,  doctrine, 
logistics,  maintenance,  and  the  postulated  threat.  The  degree  of 
operational  realism  differs  between  the  varieties  of  user  tests, 
because  each  variety  has  a  different  focus  depending  upon  its 
purpose.  Each  variety  of  operational  test  may  contribute  toward 
the  goal  of  the  E3  program  that  Army  materiel  will  accomplish  its 
intended  mission  in  the  electromagnetic  environment. 

b.  Planning.  To  support  the  E3  program,  test  planning  for 
this  category  of  user  test  should  include  scenarios  that  expose 


*  An  operational  fix  is  defined  as  a  change  in  training,  doctrine, 
and/or  deployment  such  that  the  conditions  causing  the  anomaly  to 
occur  can  be  either  avoided  or  recognized  as  a  transient  phenomena. 
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the  system  to  the  projected  threat  electronic  warfare  and  to 
friendly  emissions  from  both  other  inter-operating  systems  and 
from  systems  operating  in  the  vicinity.  In  actuality,  safety  and 
commercial  frequency  constraints  often  lead  to  test  limitations 
upon  the  degree  to  which  E3  testing  may  be  conducted.  Those 
responsible  for  planning  this  category  of  user  testing  must  start 
coordinating  as  early  as  possible  to  develop  a  test  plan 
acceptable  to  all  agencies  concerned  and  to  obtain  the  necessary 
frequency  clearances.  The  overall  Army  POC  for  frequency 
clearance  matters  is  the  Army  Spectrum  Manager  (ASM),  HQDA,  ATTN: 
SAIS-SM,  Room  2C513,  The  Pentagon,  Washington,  DC  20310. 

To  support  a  frequency  clearance  request,  the  ASM  requires  a 
proposed  test  plan,  system  engineering  explanations  to  justify 
that  test  plan,  a  postulated  threat  description,  a  complete  list 
of  frequencies  that  might  be  affected,  the  projected  degree  of 
those  effects,  and  the  geographic  distribution  of  those  effects. 
After  assimilating  this  information,  the  ASM  will  work  with  the 
test  planner  to  determine  which  E3  testing  alternatives  are 
viable.  Additional  guidance  is  found  in  Section  C,  Part  6, 

DODI  5000.2. 

c.  Resourcing  is  accomplished  in  accordance  with  Chapter  8, 

AR  73-1,  and  AR  37-100-XX.  Refer  to  Table  14-4  of  this  Section 
for  specific  test  center  resources  for  E3 ,  and  Table  14-4  for  E3 
Organizations/Off ices. 

d.  Test  execution  will  be  a  representation  of  an  approved 
scenario  in  accordance  with  AR  73-1.  Discussion  of  scenario 
development  will  be  in  each  system's  TEMP  in  accordance  with 
Part  7,  DOD  5000. 2-M,  for  AR  73  series  systems  or  Part  7, 

DOD  7920. 2-M.  Additional  guidance  on  execution  is  in  Part  V  of 

this  Pamphlet. 

e.  Reporting  guidance  is  contained  in  Part  V  of  this 
Pamphlet. 

14-13.  Continuous  Evaluation  (CE) 

a.  CE  is  a  continuous  process  extending  from  concept 
definition  into  deployment  which  evaluates  the  operational 
effectiveness  and  suitability  of  a  system  by  all  available  data. 
CE  is  addressed  in  greater  detail  in  Chapter  2,  of  this  Pamphlet. 

Planning.  As  mentioned  in  paragraph  14— 5b,  operational 
testing  in  support  of  the  E3  program  can  be  constrained  by  safety 
and  other  restrictions.  To  overcome  these  potential  limitations, 
the  independent  operational  evaluator  and  the  independent 
developmental  evaluator  must  coordinate  and  integrate  their 
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evaluation  efforts  to  ensure  E3  issues  are  adequately  addressed. 
They  will  review  the  subject  system's  COIC.  For  command, 
control,  communications,  and  intelligence  (C3I)  and  IMA  systems, 
they  will  analyze  if  E3  criteria  are  appropriate  for  the  COIC  and 
make  recommendations  to  their  commands  during  the  staffing  of  the 
COIC.  When  necessary,  the  independent  operational  evaluator  may 
develop  Additional  Operational  Issues  and  Criteria  to  support  the 
E3  Program.  The  two  primary  methods  for  integrating  E3  testing 
are: 


(1)  the  independent  operational  evaluator  and  the 
developmental  evaluator  work  through  the  TIWG  for  the  subject 
system  to  ensure  the  TEMP  has  an  adequate  plan  for  E3  testing; 

(2)  the  developmental  evaluator  provides  a  test  report 
covering  E3  developmental  testing  to  the  independent  operational 
evaluator  who  in  turns  adjusts  the  T&E  strategy  as  appropriate  to 
ensure  all  necessary  E3  issues  will  be  addressed  by  the  Milestone 
III  (Production  Approval)  decision. 

c.  Resourcing  is  jointly  accomplished  by  the  system's 
product/project  manager  (PM) ,  the  independent  operational 
evaluator,  and  the  developmental  evaluator  through  the  Test 
Schedule  and  Review  Committee  (TSARC)  process  and  the  budget 
processes  of  involved  commands.  A  TEMP  will  include  the 
resourcing  required  for  each  key  testing  activity.  The 
evaluators  must  budget  for  other  activities  through  their  own 
directorates.  Refer  to  Table  14-3  of  this  Section  for  specific 
test  center  resources  for  E3,  and  Table  14-4  for  E3 
Organizations/Offices. 

d.  Execution.  The  key  to  execution  is  gathering  information 
from  a  variety  of  sources  and  analyzing  trends  to  identify  where 
areas  of  highest  risk  to  the  Army  are.  E3  is  one  of  the  areas 
that  must  be  addressed. 

e.  Reporting.  The  preparation  of  lERs  by  both  the 
operational  and  the  developmental  evaluators  is  addressed  in 
Parts  5  and  4,  respectively  of  this  Pamphlet. 

(insert  table  14-8) 

14-4.  Test  &  Evaluation  Command  (TECOM) 

a.  Combat  Systems  Test  Activity  (CSTA) .  Located  at  Aberdeen 
Proving  Ground,  the  Electromagnetic  Interference  Enclosure  is  a 
solid  panel  design  shielded  enclosure  approximately  94 '1  x  60 'w  x 
28 'h  with  reinforced  floors  and  large  access  doors  (16'  x  16'). 
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An  air  intake  and  exhaust  system  with  a  12,000  cfm  capacity 
permits  operation  of  equipment  including  engines  within  the 
enclosure  during  testing.  All  electrical  power  and  communication 
lines  entering  the  enclosure  are  equipped  with  100  dB  in-line 
filters.  An  instrumentation  room,  a  12 '1  x  12 'w  x  8'h  solid 
shielded  enclosure,  is  used  to  house  the  measuring  and  recording 
instrumentation.  Computer-controlled  instrumentation  is 
available  for  controlling,  measuring,  displaying,  and  recording 
data  in  accordance  with  MIL-STD-461,  462,  and  463  series  on  EMI 
testing.  RF  power  sources  are  available  to  provide  the  RF 
frequencies  and  field  intensity  levels  specified  for 
MIL-STD-461C.  The  Electromagnetic  Environmental  Test  Facility 
(EMETF)  is  used  to  determine  the  ability  of  C-E  equipments  and 
systems  to  operate  in  their  intended  operational  environments 
without  suffering  or  causing  unacceptable  degradation  because  of 
unwanted  electromagnetic  radiation  or  response.  Testing 
capabilities  address  unintentional  interference  as  well  as 
vulnerability  to  electronic  countermeasures  (ECM) .  The  EMETF 
performs  laboratory  and  field  testing,  modeling  and  simulation, 
and  research  and  development  activities.  The  EMETF  has  five 
empirical  measurement  facilities,  a  mobile  field  test  facility, 
and  a  field  test  area.  The  mobile  field  test  facility  and  the 
empirical  measurement  facilities  are  used  to  determine  equipment 
characteristics  and  the  degree  of  degradation  that  different 
types  of  interference  cause  to  C-E  and  weapon  systems,  and  the 
conditions  under  which  this  occurs.  The  five  empirical 
measurement  facilities  are:  the  Instrumented  Workshop;  the 
Spectrum  Signature  Facility;  the  Voice  Scoring  Facility;  the 
Electro-Optical  Test  Operation;  and  the  Stress  Loading  Facility. 
Empirically  derived  data  are  used  in  conjunction  with  the  EMETF 
analytical  computer  facility  and  a  library  of  computer  models  to 
analyze  the  probability  of  satisfactory  operation  or  system 
effectiveness  of  tested  systems  or  equipment  when  deployed  in  a 
tactical  environment.  Analyses  are  performed  to  determine  not 
only  the  impact  of  the  electromagnetic  environment  on  equipment 
or  systems,  but  also  the  impact  of  equipment  or  systems  on  the 
environment.  The  library  of  computer  models  used  for  these 
analyses  of  C-E  equipment  and  systems  in  turn  use  realistic 
simulated  tactical  deployment  data  bases. 

(1)  Instrumented  Workshop.  The  Instrumented  Workshop 
(IWS) ,  while  primarily  used  for  link  measurement  tests,  to 
include  encrypted  links,  also  supports  exploitation  efforts.  For 
link  measurements,  the  IWS  uses  three  shielded  screen  rooms  into 
which  the  Test  Link  Transmitter  (TLT) ,  Test  Link  Receiver  (TLR) , 
and  Interface  Generator  (IG)  are  placed.  Through  the  use  of 
coaxial  cable  and  programmable  attenuators,  a  realistic 
communications  link  is  established  between  the  TLT  and  TLR.  The 
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J/S  level  is  then  altered  to  establish  the  system-under-test's 
performance  under  ECM  conditions.  The  output  data  for  each 
individual  test  can  be  printed  in  camera-ready  format  then  or 
later  for  direct  inclusion  in  a  report. 

(2)  Spectrum  Signature  (SS) .  Spectrum  signature 
measurements  of  communications-electronics  equipment 
characteristics  can  be  performed  in  accordance  with  Military 
Standard  449C,  or  as  required  by  the  specific  project. 
Measurements'  of  transmitter,  receiver,  and  antenna 
characteristics  are  performed  as  well  as  specialized  measurements 
on  missiles,  automatic  data  processors,  switching  equipments,  and 
weapon  systems.  Mobile  vans  for  field  measurement  requirements 
are  also  available.  Each  van  is  equipped  with  shielded 
enclosures  and  a  complete  range  of  signal  generators,  spectrum 
analyzers,  distortion  analyzers,  and  other  specialized 
measurement  equipments  covering  a  frequency  range  from  50  kHz  to 
40  GHz. 

(3)  Voice  Scoring  Facility.  The  Voice  Scoring  Facility 
has  the  capability  of  scoring  voice  intelligence  transfer  that 
occurred  over  clearer  degraded  (interference  and/or  jamming) 
analog  and  digital  communications  links.  The  facility^ can  score 
carrier-phrased  phonetically-balanced  (PB)  words  and  diagnostic 
rhyme  test  (DRT)  words.  The  facility  contains  eight  listener 
console  positions  and,  in  a  separate  room,  a  control  position 
with  a  one-way  mirror  for  observing  the  listeners.  Each  listener 
console  contains  a  video  monitor,  a  touch  wand,  a  high-fidelity 
headset,  and  a  volume  control.  The  control  console  contains  a 
personal  computer  (PC)  and  a  high-fidelity  playback  machine.  The 
PC  with  its  programs  is  used  to  put  words  to  be  scored  on  the 
video  monitors.  Listeners'  video  monitors  are  touch-screen 
sensitive.  Listeners  touch  the  word  (that  they  think  they  heard) 
on  their  touch  screens  with  their  touch  wands.  PB  words  are  used 
in  the  analysis  of  analog  circuits.  DRT  words  are  used  in 
systems  developments  and  in  digital  circuits.  A  very  high  degree 
of  repeatability  has  been  demonstrated  over  periods  as  long  as 
five  years. 

(4)  Electro-optical  Testing  Facility.  Tests  of  the 
equipment  can  be  made  both  in  the  laboratory  and  in  the  field 
over  the  wavelength  interval  from  ultraviolet  to  the  far 
infrared.  Currently,  methodologies  and  test  equipment  are  being 
developed  to  test. imaging  Automatic  Target  Recognition  systems. 

In  this  case,  the  sensors  can  be  television,  forward-looking 
infrared  and  synthetic  aperture  radar.  An  emphasis  on  sensor 
fusion  will  allow  EMC  and  EMV  testing  of  military  hardware  using 
integrated  data  from  multiple  sensors. 
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(5)  Stress  Loading  Facility.  The  Stress  Loading  Facility 
(SLF)  bridges  the  gap  between  field  tests  and  software 
simulations  by  providing  a  realistic  RF  environment  with 
controlled  and  repeatable  conditions.  The  SLF  provides  a 
low-level  RF  environment  in  a  closed  facility  which  limits 
emanation  of  RF  signals  from  the  test  environment  while 
preventing  contamination  from  the  external  RF  environment.  The 
SLF  can  be  used  to  test  a  system  below,  at,  or  above  its 
specif ication- limits.  The  SLF  consists  of  three  subsystems: 

(a)  The  Communications  Threat  Simulator  (CTS)  is 
capable  of  generating  a  wide  variety  of  waveforms  from  voice, 
digital,  and  jamming  to  commercial  television  in  the  0.5  to  500 
MHz  band.  The  CTS  consists  of  32  signal  sources  (expandable  up 
to  128  sources) . 

(b)  The  Non-Communications  Threat  Simulator  (NCTS)  can 
simulate  up  to  1024  dynamic  pulsed  emitters  on  a  time-multiplexed 
basis.  The  emitters  simulated  include  pulsed  radar,  navigational 
and  other  non-communications  signals  in  the  0.5  to  18  GHz  band 
(with  future  enhancements  up  to  40  GHz  and  selected  higher 
frequencies) . 

(c)  The  Functional  Systems  Simulator  (FSS)  is  a 
software  and  hardware  subsystem  which  simulates  ancillary 
hardware  or  software  processing  for  the  system  under  test. 
Functions  of  the  FSS  include  simulating  support/ control  systems 
associated  with  the  test  system,  monitoring  system  performance, 
generating  scenarios  and  running  predesignated  scenarios. 


(d)  Mobile  Field  Test  Facility.  The  Mobile  Field  Test 
Facility  consists  of  jamming,  intercept,  control,  and  monitoring 
equipments  which  are  used  to  gather  data  to  address  the  issues  of 
specific  tests.  Equipments  are  configured  into  three  transmitter 
and  two  receiver/monitor  vans  which  can  be  utilized  at  a  variety 
of  test  sites.  Additionally,  the  EMETF  has  six  modular 
palletized  generic  jammer  systems  for  either  airborne  or  ground 
use.  The  six  modular  palletized  jammer  systems  are  currently 
configured  to  operate  from  a  EH/UH-H  helicopter  or  to  be 
trailer-mounted  for  ground-based  use.  Various  modulation  types 
are  available,  including  CW  comb,  Gaussian  noise,  or  any  waveform 
which  can  be  generated  by  a  waveform  synthesizer.  Operation  of 
the  palletized  jammers  is  computer  controlled  via  the  jammer 
operator's  keyboard/monitor.  Jammers  can  be  programmed  for 
specific  effective  radiated  powers,  transmission  times,  comb 
generator  frequencies  enabled,  calibration  statuses,  and  RF  power 
outputs  versus  times.  The  performance  of  the  jammer  can  be 
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automatically  monitored  and  recorded.  The  EMETF  has  three 
semi-trailer,  van-mounted  generic  jamming  systems.  The  jamming 
systems  can  generate  a  wide  variety  of  waveforms  for  radar 
simulation,  radar  jamming,  communications  simulation,  and 
communications  jamming.  Power  amplifiers  and  antennae  are 
available  covering  from  0.5  MHz  to  18  GHz.  Equipment  can  be 
configured  and  operated  automatically,  remotely,  or  manually. 

Each  van's  antennae  can  be  accurately  positioned  from  0  to  360 
degrees  in  azimuth  and  to  +45  degrees  in  elevation.  The  two 
EMETF  intercept/direction  finding  vans  are  semi-trailer  mounted 
generic  intercept/DF  systems.  The  receiving  systems  can  be 
controlled  manually,  via  console  keyboard,  or  by  computer  for 
automatic  functions  of  intercept  and  DF.  A  variety  of  search 
option  modes  are  available.  In  all  modes,  the  system  is  capable 
of  automatically  determining  received  signal  strength,  signal 
frequency,  pulse  width,  pulse  repetition  interval  (PRI) ,  pulse 
repetition  frequency,  average  pulse  width,  average  PRI,  and  angle 
of  arrival.  Recording  of  individual  receiver  video  output, 
timing  signals,  and  command  and  control  communications  is 
possible  during  test.  The  receiving  systems  cover  from  1  to  18 
GHz.  The  antennae  can  cover  from  1  to  40  GHz.  One  van  has 
automated  receiving,  signal  analysis,  and  DF  capabilities. 

(e)  Field  Facility.  The  field  test  area  consists  of 
5000  acres  checker  boarded  within  a  40  x  60  mi  area.  It  may  be 
used  to  test  all  types  of  communications  and  electronic  systems 
under  open  field  conditions.  The  extremely  low  level  of 
background  radio  and  electronic  signals  enables  equipments  to  be 
tested  and  evaluated  under  controlled  interference  conditions 
without  background  signals  which  could  bias  measurements. 

(f)  Analytical  Facility.  The  EMETF  uses  computer-based 
modeling  to  simulate  and  analyze  EMC,  EMV,  and  performance  of  C3I 
test  items.  Model  simulations  and  computer  analyses  are 
conducted  on  a  CDC  CYBER  180  model  830  computer  system  housed  in 
a  secure  facility.  Model  development  is  enhanced  through  the  use 
of  the  Simulation  Language  for  Analysis  of  Communications  Systems 
(SLACS)  developed  by  the  EMETF.  An  extensive  library  of  software 
models  allows  the  EMETF  to  simulate  test  item  C-E  functions,  and 
to  support  battlefield  scenario  activities,  threat  electronic 
warfare,  and  intelligence  activities.  The  EMETF  can  draw  from 
applicable  portions  of  these  models  to  support  specific  customer 
requirements.  Models  contain  propagation  algorithms  for 
calculating  signal  and  performance  levels  for  each  equipment 
under  various  propagation  modes,  terrain  conditions,  and 
distances.  Empirical  data  on  the  performance  of  the  equipments 
or  systems  under  varying  conditions  of  intentional  or 
unintentional  interference,  simulated  tactical  deployment  data 
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bases  of  the  tactical  environment,  and  parametric  data  bases 
drive  models  during  simulated  runs.  The  simulated  tactical 
deployment  data  bases  are  developed  in-house  to  meet  customer  and 
threat-community  specifications,  and  include  RED,  BLUE,  and  GRAY 
elements  as  appropriate.  Simulated  tactical  deployments  have 
varied  in  size  from  small  to  echelons-above-corps  (EAC) .  Model 
outputs  are  detailed  histories  of  simulation  activity,  test  item 
operation,  and  model  processes.  Post-processing  routines  use 
detailed  history  files  to  generate  reduced  data  and  statistical 
summaries  to  analyze  test  item  EMC,  EMV,  and  performance. 

Modeling  and  simulation  analyses  permit  customers  to  evaluate 
system  designs,  product  improvements,  deployment  and  operations 
concepts,  equipment  mix  options,  mutual  interference  issues,  and 
vulnerability  concerns.  The  Computer  Automated  Analysis 
Capability  uses  a  library  of  computer  models  in  conjunction  with 
a  multiple  user  CDC  CYBER  180  model  830  computer  to  perform 
electromagnetic  compatibility  and  vulnerability  analyses  of 
communications-electronics  concepts,  systems,  and  equipments  in 
typical  tactical  environments.  This  library  of  computer  models 
consists  of  computer  programs  and  routines  for  use  in  a  variety 
of  electromagnetic  compatibility  evaluations.  An  analysis  model 
for  a  specific  task  is  constructed  by  selecting  from  the  library 
of  programs  of  the  routines  necessary  to  perform  the  particular 
task  analysis. 

(g)  Electromagnetic  Interference/Tempest  Test  Facility. 
The  Blacktail  Canyon  EMI /TEMPEST  facility  is  located  in  a  remote 
RF  isolated  area  of  Ft.  Huachuca.  The  remote  location  provides  a 
relatively  low  electromagnetic  ambient  environment  which 
optimizes  open-field  testing.  The  facility  location  in 
conjunction  with  a  400  ft  by  360  ft  perimeter  fence  provides  the 
degree  of  physical  security  required  for  mission  tests.  Testing 
can  be  accomplished  in  accordance  with  the  following  standards: 
EMI  (MIL-STD-461C  and  MIL-STD-462) ;  TEMPEST  (NACSIM  5100A, 

NACSIM  5112  and  KAG  30) ;  and  lEMC  (MIL-STD-6051) . 

(1)  Three  EMI /TEMPEST  test  chambers  include:  a  44  ft  long 
by  22  ft  wide  by  18  ft  high  anechoic  chamber  which  provides  120 
db  of  RF  isolation  and  will  accommodate  military  equipment  up  to 
the  sizes  of  the  HMMWV,  CUCV,  LAV,  and  M113  families;  a  26  ft 
long  by  16  ft  wide  by  11.5  ft  high  TEMPEST/EMI  chamber  providing 
100  db  RF  isolation;  and  a  12  ft  long  by  10  ft  wide  by  11  ft  high 
shielded  room  for  testing  of  small  items. 

(2)  Facility  instrumentation  suites  consists  of  the 

following:  two  Dynamic  Sciences,  Inc.  TEMPEST  test  systems 

providing  automatic  NACSIM  and  KAG  testing  requirements;  two 
automated  AILTECH  RFI/EMI  data  collection  systems  providing 
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support  to  MIL-STD-461C/462  radiated  and  conducted  emission 
testing  from  20  Hz  to  40  GHz;  an  integrated  EMI  susceptibility 
system  allowing  RF  illumination  of  equipment  from  lOKHz  to  40 
GHz;  and  an  extensive  assortment  of  parallel  element,  rod, 
biconical,  log  periodic,  and  double  ridge  guide  antennae,  along 
with  associated  RF  amplifiers  and  electric  field  probes  which  can 
provide  RF  illumination  and  detection  capabilities  across  the  40 
GHz  spectrum  relevant  to  the  EMI/TEMPEST  arena. 

(3)  The  EMI/Rab  data  collection  and  TEMPEST  systems 
provide  sufficient  portability  to  allow  performance  of 
EMI /TEMPEST  tests  at  remote  locations.  Remote  TEMPEST  testing  is 
also  accommodated  with  two  mobile  vans.  One  van  is  equipped  with 
a  Watkins- Johnson  manual  TEMPEST  measurement  system.  The 
remaining  van  houses  a  DSI  9000  seri  automated  TEMPEST 
measurement  system. 


(h)  The  Test  Item  Simulator  (TIS)  is  a  computer-based 
test  driver  used  to  simulate  C3I  systems  in  a  test  environment. 
The  TIS  generates  and  records  a  data  environment  which  exercises 
and  stresses  systems  under  test  to  measure  their  performance 
(system,  software,  and  interoperability).  The  TIS  provides 
controlled  and  repeatable  technical  and  operational  simulation 
representing  multiple  C3I/IEW  systems  to  the  system(s)  under  test 
and  monitors  and  records  their  responses  in  real-time.  It 
represents  tactical  systems  by  transmitting  a  controlled  and 
repeatable  message  stream  using  the  tactical  system  message 
formats  and  protocols.  The  TIS  consists  of  both  small  and  large 
capacity  systems.  The  large  system  is  mounted  in  two  40  foot 
semi-trailer  vans  while  the  small  version  is  installed  in  a 
ruggedized  case  for  operation  in  a  field  environment.  The 
Antenna  Measurement  Facility  is  used  to  determine  radiation 
patterns  from  both  mounted  and  unmounted  antennas.  The  facility 
consists  of  a  114-foot  nonmetallic  tower,  177-foot  sensor-bearing 
arc,  and  two  rotating  turntables  30  feet  in  diameter,  one  under 
the  arc  and  the  other  500  feet  east  of  the  tower.  The  facility 
is  used  for  testing  electronic  equipment  or  antennas  from  0  to 
100  feet  above  the  ground.  The  arc  is  an  antenna  measurement 
facility  and  makes  possible  the  determination  of  antenna  patterns 
of  various  radiating  devices  while  they  are  in  operational 
position  on  the  aircraft.  The  arc  is  constructed  of  laminated 
wood  and  has  essentially  free  space  characteristics  above  1000 
MHz.  In  addition  to  the  arc  with  its  75  foot  radius  and  the 
turntable,  this  system  includes  a  rotatable  platform  which  can 
support  inverted  aircraft  airframes  (up  to  10  tons  weight) .  The 
facility  is  equipped  with  instrumentation  for  the  evaluation  of 
antenna  systems  from  50  MHz  to  18  GHz.  This  facility  is  used  to 
determine  radiation  patterns  from  both  mounted  and  unmounted 
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antennas  and  complements  the  Antenna  Measurement  Facility.  The 
usable  frequency  range  is  from  6  GHz  to  40  GHz,  with  potential 
increase  to  95  GHz.  The  positioner  can  hold  test  objects  (such 
as  aircraft  and  ground  vehicles)  weighing  up  to  70  tons,  42  feet 
above  the  ground,  and  rotate  the  test  object  in  both  azimuth  and 
elevation.  Five  different  feed  horns  cover  the  frequency  ranges: 
6-8  GHz,  8-12  GHz,  12-18  GHz,  18-26.5  GHz,  and  26.5-40  GHz.  The 
75  foot  diameter  parabolic  reflecting  antenna  produces  a 
collimated  beam  with  a  50  foot  diameter  flat  wave  front  (or  quiet 
zone)  . 


c.  Redstone  Technical  Test  Center  (RTTC) 

(1)  Electromagnetic  Interference  (EMI)  Test  Facility.  The 
EMI  test  facility  consists  of  a  13  foot  by  30  foot  double 
shielded,  copper  screen  room,  divided  into  a  test  and  a  control 
room.  The  facility  is  capable  of  measuring  emissions  and 
susceptibilities  during  subsystem  and  system  tests  as  specified 
in  MIL-STD-461  and  MIL-STD-462.  All  measurement  equipment  is 
periodically  calibrated  and  the  major  components,  such  as 
frequency  synthesizers,  spectrum  analyzers,  and  power  meters,  are 
controlled  by  an  HP  1000-A400  computer  to  enable  rapid 
quantitative  measurements  to  be  obtained,  recorded,  and  plotted. 
To  ensure  that  there  are  not  problems  when  assembled  into  a 
weapon  system,  items  may  be  tested  to  determine  the  effects 
between  subsystems,  effects  of  subsystems  upon  external  systems, 
and  effects  of  external  systems  upon  the  subsystem. 

(2)  Electrostatic  Discharge  (ESD)  Facility.  ESD  testing 
of  components  or  systems  is  conducted  to  verify  the  safety  and 
survivability  of  those  items.  Survivability  is  assessed  by 
proper  system  operation  following  exposure.  Safety  is  determined 
through  a  set  of  go/no-go  tests  or  through  instrumented  tests  and 
comparison  to  a  database.  Go/no-go  safety  tests  typically  focus 
on  munition  firing  circuits,  particularly  the  EEDs  or  other  types 
of  initiators.  In  these  one-shot  tests,  the  actual  EEDs  are 
installed  in  the  munition  to  be  tested.  The  failure  criteria  is 
the  functioning  of  the  device  or  a  change  in  a  critical 
parameter,  such  as  the  resistance  of  the  bridgewire. 

Instrumented  tests  require  the  development  of  sensor  packages  to 
replace  the  EEDs  and/or  to  monitor  firing  circuit  test  points, 
such  as  within  a  S&A  subassembly.  High  frequency  fiber  optic 
data  links,  transient  digitizers,  and  computer  controller  and 
analyzer  equipment  are  used  to  acquire  and  analyze  the  needed 
data.  Ion  Physics  Corp.  electrostatic  generators  capable  of  0  -+ 
30  kv  and  0  -+  300  kv,  and  HIPOTRONIC  8000  Series  Power  Supplies 
and  high  voltage  capacitor  banks  are  used  for  personnel  borne  and 
helicopter  borne  tests.  These  tests  can  be  conducted  on  both 
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inert  and  live  tactical  missile  system  hardware. 

(3)  Electromagnetic  Pulse  (EMP)  Facility.  The  EMP  test 
facility  is  used  to  conduct  tests  simulating  the  high  altitude 
EMP  waveform  as  defined  in  DOD-STD-2169A  and  Quadripartite  ^ 
Standardization  Agreement  (QSTAG)  244,  Edition  3.  The  facility  is 
a  radiated  freefield  type  simulator  and  consists  of  a  100  kV 
pulser,  10  meter  diameter  horizontally  polarized  dipole  antenna 
305  meters  long,  power  distribution  system,  pulser  control 
station,  and  shielded  instrumentation  van.  System  peculiar 
instrumentation,  fiber  optic  data  links,  transient  digitizers, 
and  computers  are  used  to  acquire  and  analyze  system  EMP  effects. 
EMP  testing  is  conducted  to  determine  weapon  system  safety  and 
survivability  and  is  usually  performed  at  the  system  level. 

(4)  Electromagnetic  radiation  hazard  (EMRH)  and 
electromagnetic  radiation  operational  (EMRO)  facilities.  EMRH 
and  EMRO  system  test  criteria  are  contained  in  MICOM  Technical 
Report  RD-TE-87-1.  These  criteria  are  based  on  analyses  of  the 
electromagnetic  environments  for  Army  deployments.  Navy  shipboard 
environments,  and  North  Atlantic  Treaty  Organization  (NATO) 
requirements.  These  criteria  are  periodically  reassessed  to 
accurately  reflect  the  evolving  environment.  Detailed  mission 
definitions  are  used  to  tailor  these  criteria  for  specific  weapon 
systems . 

(a)  Broadband  transmitters  are  utilized  for  Continuous 
Wave  (CW) ,  Amplitude  Modulation  (AM) ,  Frequency  Modulation  (FM) , 
and  Pulse  Modulation  (PM)  testing  with  several  subsets  of 
antennas  covering  the  100  kHz  to  18  GHz  frequency  range. 

(b)  Future  expansion  includes  fabrication  of  a  40  feet 
wide,  70  feet  long,  22  feet  high  anechoic  chamber  for  testing 
weapon  systems  and  components.  This  facility  includes  a  full  360 
degrees  of  rotation  turntable  capable  of  accommodating  a  tracked 
vehicle  the  size  of  a  M-270  launcher  (MLRS) .  The  frequency 
coverage  of  this  test  chamber  will  be  100  MHz  to  40  GHz. 

(c)  EMRH  testing  requires  instrumentation  of  the  firing 
circuits  and  electric  pyrotechnic  initiators.  Since  missiles  are 
designed  in  a  wide  variety  of  configurations  and  use  a  number  of 
different  initiators,  calibrated  instrumentation  is  a  significant 
design  problem  for  each  test  program.  The  EM&NE  Test  Function 
has  an  established  capability  for  accurate  instrumentation 
design,  fabrication,  and  integration  so  that  the  instrumentation 
does  not  introduce  additional  ports  of  entry  for  EMR  into  the 
system.  This  capability  results  in  reproducible  test  data  and 
provides  the  basis  for  accurate  safety  margin  calculations. 
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(d)  Since  the  coupling  of  the  system  is  a  function  of 
its  orientation  and  the  characteristics  of  the  EMR  environment, 
extensive  testing  is  required  to  assess  the  electromagnetic 
environmental  effects.  System  operation  assessment  requires 
parameter  extraction  from  the  system,  as  well  as  equipment  used 
to  simulate  the  mission.  Again,  the  established  capability  of 
the  EM&NE  Test  Function  to  design,  build,  and  install  hardened, 
fiber  optic  coupled  telemetry  provides  a  timely,  cost  effective 
approach  to  EMRO  instrumentation. 

(5)  Lightning  test  facilities.  Testing  for  evaluating  the 
effects  of  lightning  may  be  divided  into  two  categories,  direct 
strike  and  close  lightning  (i.e.  magnetic  and  electric  fields). 
Test  criteria  are  derived  from  MIL-STD-1757 ,  Lightning 
Qualification  Test  Techniques  for  Aerospace  Vehicles  and 
published  in  MICOM  Technical  Report  RD-TE-87-1.  Lightning 
simulation  generators  capable  of  generating  up  to  3.6  million 
volts  and  200,000  amperes  are  used  for  these  tests. 

(a)  Direct  strike  test  criteria  are  primarily  required 
for  weapon  system  safety  and  secondarily  required  to  prevent 
permanent  damage  to  system  electronic  components.  Tests  are 
conducted  to  verify  that  the  system  is  safe  from  premature  launch 
or  detonation  of  hazardous  items  when  subjected  to  a  direct 
lightning  strike. 

(b)  Close  lightning  strikes  sometimes  referred  to  as 
Lightning  Electromagnetic  Pulse  (LEMP)  criteria  are  required 
primarily  for  protection  of  EEDs  and  electronic  components  from 
detonation,  burnout,  destruction,  etc.  Large  magnetic  and 
electric  fields  are  radiated  from  lightning  strikes,  and  it  is 
necessary  to  design  missile  systems  to  withstand  these 
environments  during  a  launch  sequence  or  when  the  electronics  are 
active. 


(c)  Testing  is  conducted  on  both  inert  and  live, 
tactical  missile  systems.  Testing  is  also  conducted  by  go/no-go 
testing  of  actual  EEDs  or  squibs  or  by  using  instrumented  devices 
to  measure  transient  currents  and  voltages  induced  in  the 
bridgewire  of  each  test  item. 

(d)  The  RTTC  lightning  test  capabilities  consist  of 
three  distinct  test  facilities.  The  Inert  Lightning  Test 
Facility  is  utilized  for  instrumented  and  go/no-go  testing  of 
systems  limited  to  Class  1.4  explosives.  The  Hazardous  Lightning 
Test  Facility  is  comprised  of  two  facilities  which  are  tailored 
for  test  object  size  and  explosive  quantity.  The  Small  System 
Lightning  Test  Stand  is  utilized  for  testing  live,  tactical  man- 
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portable  and  other  small  missile  systems.  The  Large  System 
Lightning  Test  Stand  is  utilized  for  testing  larger  live, 
tactical  missile  systems  and  is  currently  limited  to  100  pounds 
of  Class  1.1,  5,000  pounds  of  Class  1.2,  15,000  pounds  of  Class 
1.3,  and  unlimited  Class  1.4  explosives. 

(e)  The  Lightning  Test  Facilities  at  RTTC  are  unique  in 
that  they  are  the  Army's  only  lightning  test  facilities.  The 
Hazardous  Lightning  Test  Facility  is  especially  unique  in  that  it 
is  the  only  f-acility  in  the  free  world  capable  of  testing  live, 
tactical  missile  systems  to  the  lightning  environment.  This 
facility  also  has  a  portable  environmental  conditioning  chamber 
capable  of  conditioning  tracked  vehicles  to  both  "Hot"  and  "Cold" 
temperature  extremes . 

d.  WHITE  SANDS  MISSILE  RANGE  (WSMR) 

(1)  WHITE  SANDS  EMP  SYSTEMS  TEST  ARRAY  (WESTA) .  The  White 
Sands  EMP  Systems  Test  Array  (WESTA)  provides  a  test  environment 
simulation  of  an  exoatmospheric  nuclear  weapon  detonation.  The 
facility  is  a  bounded  wave  type  with  the  EMP  environment 
radiating  into  a  working  volume  13.4  m^  and  variable  in  height 
between  6.7  m  and  15.5  m.  The  system  uses  a  unique  array  type 
antenna  which  combines  elements  of  both  bounded  wave  and  free 
field  generators.  The  array  is  made  up  of  54  antenna  modules  in 
a  3  X  18  configuration,  each  module  consisting  of  double  density 
aluminum  screens  arranged  to  form  a  dihedral  horn.  Each  horn 
radiates  like  a  free  field  radiator  and  the  array,  as  a  whole, 
provides  the  field  uniformity  and  high  field  strengths  associated 
with  bounded  wave  systems.  WESTA  generates  a  horizontally 
polarized  plane  wave  with  the  pointing  vector  perpendicular  to 
the  ground.  This  configuration  provides  the  most  stringent  test 
of  ground  based  systems  where  the  ground  reflection  is  an 
important  factor. 

(a)  WESTA  produces  a  double  exponential  pulse  with  a 
risetime  on  the  order  of  10  ns,  a  1/e  decay  time  of  280  ns  and  a 
pulse  duration  of  740  ns.  The  free  field  peak  E-field  amplitude 
is  variable  between  100  and  approximately  50,000  V/m.  EMP  fields 
ylthin  any  arbitrary  horizontal  plane  are  uniform  to  within  5-5  of 
the  mean  field  intensity  except  near  the  dihedral  horns  and  the 
edges  of  the  array.  Test  level  reproducibility  is  within  3%. 
Maximum  pulse  repetition  rates  are  output  level  dependent  varying 
from  12  to  48  pulses  per  second  below  7,000  V/m,  one  pulse  per 
minute  up  to  20,000V/m,  30  pulses  per  hour  up  to  15  pulses  per 
hour  at  higher  output  levels. 
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(b)  An  optical  fiber  analog  data  transmission  system  is 
available  for  transmitting  data  from  the  exposure  volume  to  the 
instrumentation  building  where  real-time  digital  acquisition  and 
processing  are  performed  and  hard  copy  of  the  results  made 
available  within  seconds.  Pulse  response  time  for  this  system  is 
1.5  ns  with  better  than  1%  linearity  over  its  input  range. 
Facility  instrumentation  can  measure  both  system  response  and 
field  strengths.  A  variety  of  bulk  current  and  differential 
voltage  probes  is  available  for  system  response  measurements, 
while  B-dot  and  D-dot  sensors  are  available  for  test  environment 
characterization . 

(2)  ELECTROMAGNETIC  RADIATION  EFFECTS  TEST  FACILITY 
(EMRE) .  The  Electromagnetic  Radiation  Effects  Test  Facility,  is 
located  approximately  8.5  miles  north  of  the  WSMR  cantonment 
area.  A  wide  range  of  electromagnetic  testing  can  be 
accomplished  in  an  outdoor  environment,  in  a  60  ft  x  60  ft  x  12 
ft  shielded  enclosure,  or  in  a  20  ft  x  14  ft  x  12  ft  anechoic 
chamber.  For  outdoor  testing,  EMRE  has  four  testing  areas  (two 
of  which  are  provided  with  30  ft  diameter  turntables  with 
capacity  of  120,000  pounds)  and  seven  transmitters  (with  CW,  AM, 
FM  and  pulse  modulation)  which  can  provide  field  intensities  in 
excess  of  200  V/m  from  100  KHz  through  18  GHz.  Sweer 
capabilities  are  provided  in  the  ranges  of  20  KHz  through  225  MHz 
and  500  MHz  through  18  GHz.  Capabilities  and  facilities  exist 
for  development,  fabrication,  and  installation  of  necessary 
instrumentation.  In  addition,  electrostatic  discharge  facilities 
capable  of  testing  up  to  400,000  volt  levels  are  available  for 
simulation  of  discharge  of  electrostatic  energy  which  may  be 
developed  on  aircraft  and/or  personnel. 

(a)  The  Electromagnetic  Interference  (EMI)  Test 
Facility,  part  of  the  EMRE  Test  Facility,  contains  a  20  ft  x  14 
ft  X  10  ft  anechoic  chamber  and  a  60  ft  x  60  ft  x  12  ft  shielded 
chamber.  Test  criteria  and  test  instrumentation  are  in 
accordance  with  MIL-STD-461.  The  EMI  facility  has  signal 
generators,  receivers,  and  antennas  which  cover  the  frequency 
range  from  20  Hz  to  18  GHz. 

(b)  An  additional  capability  of  the  EMRE  Facility  is 
the  Antenna  Measurement  and  Radar  Cross  Section  Measurement 
Facility.  This  facility  is  used  to  determine  radiation  patterns 
from  both  mounted  and  unmounted  antennae.  It  consists  of  a  300 
ft  tower  with  an  elevator  and  a  30  diameter  turnable  with  a 
capacity  of  120,000  pounds.  The  tower  is  used  to  perform  antenna 
measurement  and  radar  cross  section  measurements  with  look-down 
angles  of  up  to  60  degrees.  This  capability  permits  EW 
assessments  and  signature  measurements  on  large  targets  while 
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rotating  them  through  360  degrees  in  azimuth. 

(3)  The  Countermeasures  Waveform  Control  System  (CWCS) 
provides  controlled  stimuli  to  radar  or  communications  receivers 
and  records  and  analyzes  their  responses.  The  system  operates  at 
frequencies  between  500  MHz  and  18  GHz  in  either  continuous  or 
pulsed  mode  with  a  duty  cycle  ranging  from  0  to  almost  100 
percent  and  is  capable  of  frequency,  amplitude,  or  phase 
modulation.  The  system  has  extensive  computer-driven  control  and 
analysis  capabilities. 

(4)  The  Multi-Purpose  Monitoring  System  (MPMS)  consists  of 
mobile  instrumentation  vans  for  monitoring  the  RF  environment 
during  ECM  and  ECCM  testing  of  air  defense,  fire  support  and  C3I 
systems.  The  MPMS  has  direction  finding  antennae,  receivers,  and 
associated  instrumentation  to  measure  and  document  the  RF 
environment  from  100  KHz  to  18  GHz  in  terms  of  frequency,  pulse 
width,  and  direction  to  the  RF  source.  Two  mobile  systems  are 
currently  available. 
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Effectiveness  1 
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Figure  14-1.  Elements  of  Survivability 
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PROBABILITY  OF  BEING  HIT  WHEN  ENGAGED 
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Against  Conventional  Weapons 
**OverlaDS  with  Mission  Performance 

Figure  14-3.  Eacaaiple  Cooeiderationa  for  Estimating  the 
Prohebility  of  Being  Hit  When  Engaged 
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^Measure  is  applicable  primarily  to  ballistic  threats  aginst  conventional 
weapons  systems. 


rigtire  14-4.  Exaxple  Influences  on  the  Probability  of 

Being  Killed  When  Hit 


SURVIVABILITY  TESTING:  VULNERABILITY  vs  AVOIDANCE 


Figure  14--5.  Survivability  testing  emphasis. 


Figure  34-6.  Survivability  Evaluation  Phases 


Critical  Issue;  How  well  does  the  system  operate  in  its 
intended  E3  environment? 

Scope;  This  issue  will  examine  the  effects  of  various 
Electromagnetic  Environments  (EME)  on  the  ability  of  the 
system  to  be  transported,  stored,  and  operated.  This  issue 
is  concerned  with  the  effects  of  unintentionally-coupled  EM 
radiation  (as  opposed  to  intentionally  directed  EM)  that  the 
system  is  expected  to  encounter.  The  E3  critical  issue  is 
comprised  of  six  sub-issues: 

Electromagnetic  Interference  (EMI) 

-  Electromagnetic  Radiation  Operational  (EMRO) 

-  Electromagnetic  Radiation  Hazard  (EMRH) 

Electrostatic  Discharge  (ESD) 

Electromagnetic  Pulse  (EMP) 

Lightning  Effects  (LE) . 

Criteria:  The  system  will  comply  with  E3-related  design 
standards/requirements,  MIL-STD  requirements  as  they  pertain 
to  the  system,  and  the  operational  electromagnetic  environment 
as  determined  by  the  E3  Requirements  Board  (E3RB) . 

Methodology;  The  system  will  undergo  E3  testing  to 
determine  compliance.  Anomalies  will  be  assessed  for  their 
impact  on  the  system's  operation. 


Figure  14-7.  An  example  of  an  E3  entry  in  the  AMSAA  lEP 
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Figure  14-9.  E3  Test  Criteria,  Test,  and  Test  Evaluation 
Overview 
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*  Pe  can  be  conceptualized  as  the  appropriate  combinations  of  factors  such  as  the 
probabilities  of  detection;  acquisition,  given  detection;  location,  given  acquisition; 
and  tracking,  given  location;  combined  with  threat  doctrine. 


Table  34-1.  Survivability  Ftinctions  and  Measures 
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Table  l4-2 .  Sample  Engagement  Avoidance  Factors 

and  Conditions 


Tabltt  14-3 

E3  Test  Selection  Methodology 


PRIMARY  E3  DOCUMENTS 


Subsystem 

or 

Component 

Aircraft 

Ground 

Vehicles 

Tactical  or 

Special 

Purpose 

Aviation 

General  Information 

EMC/ EMI 

EMV 

ESD 

Lightning 

NEMP  (EMP) 

RADHAZ 

Tempest 

ADS-37 

ADS-37 

ADS-37 

ADS-37 

ADS-37 

ADS-37 

ADS-37 

ADS-37 

Non- 

EMRO 

TOP-1-2-511 

Aviation 

EMRH 

TOP-1-2-511 

Lightning 

TOP-1-2-511 

Static  Electricity 

TOP-1-2-511 

EMP 

MIL-STD-2169 
Q-STAG  244 

General  Information 
Conducted  Emissions 
Conducted 

Susceptibility 
Radiated  Emissions 
Radiated  Susceptibility 


General  Information 
Conducted  Emissions 
Conducted 

Susceptibility 
Radiated  Emissions 
Radiated  Susceptibility 


Ordnance 


MIL-STD-461 

MIL-STD-461 

MIL-STD-461 

MIL-STD-461 

MIL-STD-461 


MIL-STD-461 

MIL-STD-461 

MIL-STD-461 

MIL-STD-461 

MIL-STD-461 


MIL-STD-461 


MIL-STD-1385B 


Table  14- 4.  Army  E3  testing  capabilities. 

TEST  CAPABILITY  SUPPORT  MATRIX 

ORGANI ZATION/ ACTIVITY 


CATEGORY 

CSTA 

EPG 

RTTC 

WSMR 

(ARMTE) 

EMC 

M 

M 

M 

M 

EMI 

M 

M 

M 
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EMRH 

M 

M 

M 

EMRO 

M 

M 

M 

EMV 

M 

ESD 

M 

M 

M 

HERO 

S 

LIGHTNING 

M 

M 

=  major  capability 

S 

=  supporting 

capability 

j  tA  ^  $'0 


Table  14-5 


E3  OR6ANIZATION8/OFFICB8 


1  AIR  FORCE 

a.  Air  Force  Flight  Test  center 
Benefield  Anecboic  Facility 
Edwards  AFB,  CA  93523-5000 
DSN  527-0840 

FAX  805-277-5405 

b.  Rone  Laboratory 

E3  Research  Center 

ATTN;  RL/ERPT 
Griffiss  AFB,  NY  13441-5700 
DSN  587-2841 
FAX  315-330-7083 


c.  Air  Force  Developmental  Test  Center 
Guided  Weapons  Evaluation  Facility 
CDR  3246  Test  Wing/TFG 

Elgin  AFB,  FL  32542-5000 
DSN  872-9798 
FAX  904-882-9929 

d.  Air  Force  Developmental  Test  Center 

Preflight  Integration  of  Munitions  and  Electronic  Systems 

CDR  3246  Test  Wing/TFP 
Elgin  AFB,  FL  32542-5000 
DSN  872-9354 
FAX  904-882-9357 


2  ARMY 

a.  Test  and  Evaluation  Command 
Combat  Systems  Test  Activity 
ATTN;  STECS-EN-EI 

Aberdeen  Proving  Ground,  MD  21005 
DSN  298-3510 
FAX  410-278-7700 

b.  Armament  RD&E  Center 
Electromagnetic  Systems  Section 
ATTN;  SMCAR-AEC-IE 

Picatinny  Arsenal,  NJ  07806-5000 
DSN  880-3025 
FAX  908-724-5861 

c.  Test  and  Evaluation  Command 
Electronic  Proving  Ground 
ATTN;  STEEP-MT-E 

Ft.  Huachuca,  AZ  85613-7110 
DSN  879-4862 
FAX  602-538-4933 


d.  Test  and  Evaluation  Command 
Redstone  Technical  Test  Center 
ATTN:  STERT-TE-E-EM 
Redstone  Technical  Test  Center 
Huntsville,  AL  35898-8052 
DSN  788-2952 
FAX  205-842-9637 

a.  Test  and  Evaluation  Command 
White  Sands  Missile  Range 
ATTN:  STEWS -TE-AG 

White  Sands  Missile  Range,  NM  88002-5157 
DSN  258-2935 
FAX  505-678-3261 


3  MAVY 

a.  Naval  Air  Engineering  Center 
ATTN:  CODE  545 

Lakehurst,  NJ  08733-5112 
908-323-7050 

b.  Naval  Air  Test  Center 
ATTN:  CODE  SYOOEE 
Patuxent  River,  MD  20670 
DSN  356-4681 

e.  Naval  Air  Weapons  Center 

Weapons  Division 
CDR  NWC 

ATTN:  CODE  36254 

China  Lake,  CA  93555-6001 

DSN  437-2948 

d.  Naval  Command,  Control  and  Ocean  Surveillance  Center 

ATTN:  CODE  825 

San  Diego,  CA  92152-5081 

DSN  553-5081 

e.  Pacific  Missile  Test  Center 
CDR  PMTC 

ATTN:  CODE  4034 

Point  Mugu,  CA  93042-5000 

DSN  351-7884 

f.  Naval  Surface  Warfare  Center 
ATTN;  CODE  H  22 

Dahlgren,  VA  22448-5000 
DSN  249-8594 
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Chapter  15 

Compatibility  and  Interoperability 


15-1.  General 

The  purpose  of  this  chapter  is  to  describe  the  procedures 
necessary  for  effective  Operational  Test  and  Evaluation  (OT&E)  of 
the  compatibility  and  resulting  interoperability  of  developing 
weapon  systems.  It  defines  compatibility  and  interoperability 
(C&I) ,  outlines  procedures  for  monitoring  compatibility  testing 
prior  to  User  Test  (UT) ,  and  defines  the  procedures  for 
determining  the  elements  of  compatibility  and  the  impacts  of 
interoperability  which  require  evaluation  in  OT&E.  This  chapter 
deals  with  system  C&I  pertaining  only  to  communications  and 
electronic  functions  or  elements.  Physical  compatibility  (e.g., 
do  trailer  hitches  match)  is  normally  covered  under  mission 
performance.  Compatibility  related  to  logistical  support  is 
covered  as  part  of  logistics  (see  chapter  10) .  C&I  definitions 
are  covered  in  Figure  15-1. 

(INSERT  FIGURE  15-1) 

15-2.  Operational  evaluator  involvement  in  system  development 

a.  Objective.  The  operational  evaluator  monitors  C&I 
requirements  development  early  in  the  acquisition  process  and 
monitors  technical  testing  as  the  system  matures.  Through  this 
effort  a  determination  is  made  on: 

(1)  The  adequacy  of  C&I  testing  prior  to  UT. 

(2)  If  there  are  any  compatibility  issues  that  make  it 
inappropriate  to  go  to  UT. 

(3)  If  identified  C&I  deficiencies  need  to  be  included  in 
the  TEP  for  UT. 

(4)  If  special  facilities,  instrumentation,  and  simulators 
for  C&I  are  required  for  UT. 

b.  Principal  sources  of  information.  The  independent 
operational  evaluator  reviews  the  major  documents  which  define 
the  system's  compatibility  and  interoperability  environment  and 
monitors  the  major  events  which  produce  information  on  C&I.  He 
also  reviews  reports  prepared  by  the  developer  or  contractor  and 
attends,  when  specifically  required,  management  and  coordination 
meetings  (such  as  design  reviews  and  TIWGs)  where  C&I  problems  or 
progress  is  reported.  The  following  are  the  potential  sources  of 
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compatibility  and  interoperability  information: 

(1)  Army  Battlefield  Interface  Concept  (ABIC).  The  ABIC 
is  produced  by  the  Combat  Developer  (CBTDEV)  (usually  TRADOC)  and 
identifies  the  intra-Army,  inter-service,  and  NATO  systems 
architecture  and  associated  interfaces.  It  serves  as  the  primary 
document  which  defines  the  systems  with  which  a  developing  system 
is  expected  to  operate. 

(2)  User  Interface  Requirements  (UIRs) .  The  UIRs  are  the 
documents  developed  by  the  CBTDEV  which  identify  the  type, 
quantity,  format,  and  frequency  of  the  data  to  be  exchanged.  A 
UIR  is  developed  for  each  pair  of  interfacing  systems  and  serves 
as  the  primary  document  defining  the  users'  requirements  of  the 
interface. 

(3)  Technical  Interface  Design  Plans  (TIDPs) .  The  TIDPs 
are  the  technical  design  documents  for  each  interface.  They  are 
developed  by  the  Materiel  Developer  (MATDEV)  and  provide  the 
technical  interface  parameters,  message  formats,  message  content, 
and  implementation  requirements. 

(4)  Interface  specifications.  Interface  specifications 
are  developed  by  the  MATDEV  and  provide  detailed  technical 
engineering  information  on  system  interfaces. 

(5)  Interface  Control  Documents  (ICDs) .  ICDs  are 
developed  by  the  MATDEV  and  describe  the  physical  and  electrical 
connections,  voltage,  and  current  requirements,  and  provide 
interface  control  drawings.  Again,  ICDs  are  not  a  primary  source 
for  operational  evaluation. 

(6)  Interface  operating  procedures  (lOPs) .  lOPs  are 
developed  by  the  MATDEV  and  describe  the  man-machine  interfaces 
and  standardized  operating  procedures  for  multiple  interfacing 
systems.  For  NATO  system  interfaces,  interoperability  is  guided 
by  Standardization  Agreements  (STANAGS) . 

(7)  Operator  and  User  Handbook.  Operator  handbooks  are 
developed  in  parallel  with  the  system  by  the  MATDEV  in 
coordination  with  the  user,  and  provide  SOPs  and  user  procedures 
relevant  to  the  operation  of  the  system. 

c.  Facilities,  instrumentation,  and  simulators.  As  the  above 
documents  are  being  developed  by  their  proponent  and  reviewed  by 
the  independent  evaluator,  development  of  the  OT&E  input  to  the 
Test  and  Evaluation  Master  Plan  (TEMP)  is  taking  place.  A 
primary  focus  of  this  effort  is  the  identification  of  facilities. 
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instrumentation,  and  simulators  necessary  for  OT&E  in  general  and 
C&I  in  specific.  Facilities,  instrumentation,  and  simulators 
necessary  to  test  and  evaluate  C&I  are  long  lead  time  items  and 
require  this  advanced  planning.  Simulators,  dynamic  scenario 
stimulators,  special  instrumentation,  and  coordinating 
capabilities  at  multiple  locations  may  be  required  and  need  to  be 
identified  early  by  the  operational  evaluator.  Review  of  the 
above  documentation  and  early  development  of  a  TEP  provide  the 
vehicle  for  this  early  planning.  Required  facilities  and 
instrumentation  are  reflected  as  early  as  possible  in  the  TEMP 
and  Outline  Test  Plan  (OTP) . 

d.  Adequacy  of  the  scope  of  technical  testing.  In  order  to 
monitor  the  depth  and  breadth  of  compatibility  testing  prior  to 
UT,  the  independent  operational  evaluator  implements  the  follow¬ 
ing  process  for  tracking  the  factors  and  conditions,  related 
interacting  systems,  and  types  of  compatibility  which  require 
testing  or  other  form  of  evaluation.  The  Operational 
Requirements  Document  (ORD)  and  ABIC  enable  the  independent 
operational  evaluator  to  identify  the  interfacing  systems  and  the 
systems  for  which  interface  is  a  concern.  The  ORD  and  UIRs  are 
used  to  identify  the  factors  and  conditions  which  have  the 
potential  to  impact  system  C&I.  Examples  of  these  factors  and 
conditions  are  shown  in  Table  15-1.  Compatibility  issues  are 
identified  by  the  evaluator  based  on  review  of  the  UIRs  and  the 
description  of  the  environment  from  the  ORD.  Examples  of 
compatibility  considerations  are  listed  in  Table  15-2. 

(INSERT  TABLE  15-1) 

(INSERT  TABLE  15-2) 

(1)  The  Evaluation  Adequacy  Matrix.  An  Evaluation 
Adequacy  Matrix  as  illustrated  in  Figure  15-2  assists  the 
independent  operational  evaluator  in  planning  for  and  determining 
the  extent  to  which  technical  testing  has  validated  system 
compatibility.  The  types  of  compatibility,  factors  and 
conditions  impacting  C&I,  and  interfacing  systems  or  equipment 
types  are  organized  as  shown  in  Figure  15-2.  Results  of 
technical  testing  are  tracked  by  coding  the  cells  to  indicate 
whether  testing  has  validated  compatibility  under  the  specified 
cell  conditions  or  whether  further  testing  is  required.  The 
completed  matrix  supports  the  assessment  presented  at  the  OTRR, 
of  whether  prior  technical  testing  has  comprehensively  validated 
compatibility.  If  the  testing  has  been  inadequate  or  has 
identified  any  critical  problem  areas  affecting  the  OT&E, 
appropriate  adjustment  to  the  Test  and  Evaluation  Plan  (TEP)  or 
Detailed  Test  Plan  (DTP)  is  made.  When  appropriate. 
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compatibility  is  further  explored  in  UT. 

(INSERT  FIGURE  15-2) 

e.  Joint  Tactical  Command,  Control,  and  Communications  Agency 
(JTC3A) .  Joint  testing  of  C&I  is  managed  by  the  JTC3A  with 
principal  assistance  by  the  Joint  Interface  Test  Force  (JITF)  and 
the  Joint  Test  Element  (JTE)  of  JTC3A  at  Fort  Huachuca,  Arizona. 
Joint  testing  may  also  be  tasked  to  one  of  the  services.  JTC3A 
and  its  sub-elements  concentrate  on  the  joint  aspects  of  C&I  and 
on  interoperability  certification.  JTC3A  has  been  assigned 
responsibility  by  DOD  Directive  5154.28  for  the  development  and 
maintenance  of  a  joint  architecture,  interface  standards,  and 
interface  definitions  for  joint  or  combined  tactical  C3  systems. 
The  JTC3A  is  also  responsible  for  developing,  testing,  and 
maintaining  interface  standards  to  be  used  by  tactical  C3  systems 
in  joint  or  combined  interfaces  in  accordance  with  guidance 
provided  by  JCS  and  OSD,  and  verifying  that  such  systems  have 
properly  implemented  the  approved  interface  standards.  JTC3A  is 
a  source  of  information  for  the  evaluator  when  joint  testing  has 
been  conducted. 

15-3.  Planning  for  C&I  in  OT&E 

a.  Evaluation  Planning.  C&I  testing  is  planned  and  executed 
according  to  procedures  described  in  the  chapters  on  user 
testing.  It  involves  the  identification  of  C&I  issues.  Measures 
of  Performance  (MOPs)  and  Data  Requirements  (Drs) .  Principal 
issues  for  C&I  are  stated  in  the  Mission  Needs  Statement  (MNS) , 
TEMP,  and  Test  and  Evaluation  Plan  (TEP) .  MOPs  and  Drs  are 
developed  using  the  dendritic  approach.  Figure  15-3  is  a  typical 
dendritic  diagram  for  evaluation  of  compatibility.  The 
compatibility  functions,  measures  of  performance,  and  data 
requirements  are  developed  by  the  evaluator  in  a  manner  similar 
to  that  illustrated.  Figure  15-4  is  a  dendritic  applying  to 
interoperability.  The  measures  and  data  relate  to  the  ability  of 
the  interfacing  systems  to  exchange  useful  information  and  use 
the  exchanged  information  effectively.  Interoperability  issues, 
MOPs,  and  Drs  are  modified  by  lessons  learned  in  technical 
testing. 


(INSERT  FIGURE  15-3) 

(INSERT  FIGURE  15-4) 

b.  MOPs.  MOPs  for  compatibility  focus  upon  the  operational 
requirements  for  exchange  of  signals  and  data  in  the  proper 
format  and  at  required  rates  within  acceptable  levels  of  error 
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detection  and  correction.  C&I  data  elements  are  planned  in  the 
user  test  concept  as  an  integral  part  of  the  user  test  trials. 

In  some  cases,  the  user  test  concept  contains  provisions  for 
special  trials  or  side  tests  which  are  required  to  capture 
specific  data  not  collectable  in  standard  scenarios.  Outstanding 
operational  C&I  issues  may  require  technical  data  (e.g. ,  traffic 
volume,  and  error  rates)  to  support  their  assessment.  Most 
operational  C&I  issues,  however,  are  resolved  by  evaluating  the 
interoperability  impact  rather  than  testing  the  technical 
performance  characteristics. 

c.  Test  planning.  Compatibility  and  interoperability 
planning  is  refined  and  implemented  in  the  TEP  and  the  DTP.  If 
compatibility  issues  are  included  in  the  TEP,  details  of  test 
planning  may  need  to  be  derived  from  technical  interface 
specifications  developed  by  the  materiel  developer.  Unresolved 
compatibility  issues  are  included  as  test  objectives  in  the  UT. 
The  test  design  topics  addressed  below  are  included  as  part  of 
the  scenario  planning  and  test  control  planning  in  both  the  TEP 
and  DTP. 

(1)  Communications.  There  are  two  types  of  communications 
associated  with  C&I  testing;  those  necessary  to  manage  and 
control  the  test  process  and  those  interoperable  communications 
to  be  evaluated. 

(a)  Test  management  and  control.  Test  management  and 
control  communications  provide  for  equipment  set-up  and  checkout, 
simulation  and  scenario  execution,  and  control  of  the  C&I  testing 
process.  Additional  data  communications  and  data  quality  lines 
may  be  necessary  for  data  collection,  data  reduction,  and 
quick-look  analysis. 

(b)  Interface  communications.  The  interface 
communications  for  the  component  systems  being  evaluated  are  to 
correspond  to  the  level  of  testing.  Communication  circuits  to  be 
used  during  the  test  are  to  be  engineered  to  insure  technical 
requirements  are  met  for  data  transfer  speeds,  signal  levels, 
circuit  type,  line  terminations  and  security  requirements. 

(2)  System  certification.  Individual  Services  are 
responsible  for  insuring  that  the  technical  and  procedural 
interfaces  between  tactical  command  and  control  systems, 
communications  systems,  communications  equipment,  and  tactical 
C3  systems  and  Intelligence  systems,  comply  with  the  interface 
documentation.  The  operational  evaluator  insures  that 
certification  testing  has  been  done  and  reflects  this  in  the 
evaluation. 
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(3)  Scenario  generation  and  execution.  The  user  message 
exchanges  are  executed  in  accordance  with  the  test  scenario  to 
insure  correct  receipt  and  proper  response  to  all  messages  trans¬ 
mitted.  Simulation  and  stimulation  may  be  required  to  exercise 
fully  the  total  interoperability  of  the  interfacing  systems. 

When  scenario  stimulators  are  used,  events  and  associated 
responses  are  defined  using  the  procedures  contained  in  operator 
handbooks  and  Interface  Operating  Procedures  documents. 

(a)  Scenario  generation.  During  test  planning  and 
execution,  communications  and  environmental  simulation 
capabilities  require  integration  with  test  events  or  tactical 
scenarios.  Scenario  drivers  provide  the  capability  to  generate, 
execute,  and  record  test  scenario  execution.  Environmental 
simulation  systems  provide  a  computer-aided  capability  for 
preparation  and  generation  of  test  scenarios  representative  of 
the  Army  operational  environment.  Simulation  systems  provide  the 
capability  to  specify  and  initiate  events,  a  real-time  monitoring 
capability,  an  on-line  scenario  modification  capability,  and  a 
capability  to  collect  all  outputs  of  the  simulation  system.  Use 
of  such  scenario  drivers  and  simulators  is  often  required  in  user 
test  of  interoperable  systems. 

(b)  Scenario  execution,  monitoring,  and  control.  The 
data  collection  capability  provided  by  automated  simulators  and 
scenario  drivers  augments  the  post-test  analysis  effort.  During 
user  testing,  exchange  of  data  between  test  systems  is  monitored 
to  insure  that  test  procedures  are  executed  as  required  and  that 
a  preliminary  assessment  of  test  results  is  possible.  A  test 
monitoring  system  provides  a  computer-assisted  capability  to 
monitor  the  exchange  of  data  and  information  within  an  interface 
in  reaction  to  simulated  test  stimuli.  These  systems  allow  the 
test  directorate  to  monitor  the  real  time  progress  of  developing 
scenarios,  observe  actual  interface  status  (in  real  time)  as  it 
relates  to  the  scenario,  tailor  the  scenario  to  each  situation, 
analyze  system  reaction  to  stimuli,  analyze  system  reaction  to 
the  operational  scenario  and  record  data  for  subsequent  playback, 
reduction,  and  analysis. 

(4)  Data  collection.  A  data  recording  capability  is  often 
required  to  collect  the  amounts  and  types  of  test  data  required 
to  support  the  field  operations,  stimulation,  simulation, 
monitoring,  and  analysis  functions.  Automated  data  recording  is 
used  to  the  maximum  extent  possible.  When  feasible,  the  user 
tester  plans  for  the  capability  to  playback  the  recorded  data  for 
post-test  analysis  or  data  reconstruction.  Early  and  repeated 
coordination  with  instrumentation  support  sources  is  required  to 
ensure  efficient  data  collection.  Test  Directorate  and  Data 
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Collection  Team  coordination  provides  the  capability  for 
as-required  modification  of  standard  collection  requirements. 

(5)  Data  exchange.  With  the  physical  and  electrical 
interfaces  established,  data  from  UT  are  collected  to  confirm  the 
capability  of  the  systems  to  exchange  the  required  data  in  the 
correct  form  without  unacceptable  error.  Data  exchange 
encompasses  user  data  (messages)  using  the  required  formats  and 
transfer  protocols. 

(6)  Data  reduction.  Automated  data  reduction  will  receive 
the  recorded  data  in  the  form  of  voltages,  signals,  freq[uencies, 
messages  transmitted  and  received,  operator  actions,  associated 
time,  and  scenario  events.  Thes*;  data  are  refined  and  provided 
to  the  evaluator  in  formats  suitable  for  evaluating  the  C&I 
performance  related  to  the  test  issues.  Manual  data  reduction 
usually  is  associated  with  the  subjective  reports,  critiques,  and 
Test  Incident  Reports  (TIRs)  generated  by  the  test  and  evaluation 
personnel  as  well  as  from  post-test  debriefings.  The  data  are 
reviewed,  abstracted,  and  sorted  in  accordance  with  the  DTP. 

15-4.  Interoperability  evaluation 

a.  Interoperability  benefits.  Interoperability  benefits 
typically  manifest  themselves  in  improvements  to  system 
performance  metrics.  Decreased  time  to  perform  a  function, 
increased  number  of  target  opportunities,  and  more  precise  or 
timely  information  are  examples  of  how  interoperability  can  be 
quantified.  These  metrics  are  often  expensive  in  that  they 
require  a  base  case  against  which  to  measure  the  increase  or 
decrease  in  performance. 

b.  Interoperability  burdens.  Interoperability  also  manifests 
itself  in  a  negative  way  by  increasing  the  time  required  to  begin 
or  complete  missions.  In  addition,  interoperability  may  require 
the  handling  and  transport  of  additional  equipment,  as  well  as 
extra  operators  and  maintainers.  The  evaluator  quantifies  these 
effects  and  uses  the  metrics  produced  to  provide  a  value  judgment 
on  the  operational  effectiveness  of  the  system.  When 
appropriate,  attention  is  given  to  the  time  to  restore  lost 
interoperability  and  the  impact  of  the  loss.  As  interoperability 
is  often  provided  among  a  large  number  of  systems,  the  evaluation 
adequacy  matrix  described  in  paragraph  15-2d(l)  of  this  chapter 
is  used  to  identify  potential  problems  areas. 
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Compatibility.  Compatibility  is  defined  by  JCS  Pub  1  as  the 
capability  of  two  or  more  items  or  components  of  equipment  or 
materiel  to  exist  or  function  in  the  same  system  or  environ¬ 
ment  without  mutual  interference.  For  data  and  communications 
systems,  compatibility  describes  whether  systems  have  the 
technical  capability  to  interface  and  exchange  data.  Compat¬ 
ibility  is  tested  and  evaluated  extensively  during  technical 
testing  and  to  a  lesser  extent  in  user  testing.  The 
independent  operational  evaluator  monitors  technical  testing 
to  validate  system  compatibility  and  to  identify  problems 
relevant  to -or  impacting  on  the  anticipated  OT&E. 

Compatibility  is  addressed  in  terms  of  electrical  (or 
electromagnetic) ,  physical  and  man-machine  interfaces  and  in 
terms  of  mutual  interference.  Compatibility  concerns  also 
apply  to  the  instrumentation  and  automation  support  plans  as 
well  as  the  tested  system. 

Electrical  and  Electromagnetic  Compatibility.  Electrical 
compatibility  involves  voltage  (llOV,  220V),  current  (AC/DC), 
and  for  alternating  current,  frequency  (60  Hz,  400  Hz,  etc). 
Electromagnetic  compatibility  encompasses  the  interconnecting 
medium  for  systems  not  physically  connected  by  wire  or  cable. 
For  radio  and  visible  light  interfaces  the  basic  consideration 
is  the  frequency  of  the  transmitted  signal.  Other  factors 
which  are  considered  include  bandwidth,  frequency  hopping 
patterns,  and  signal  polarization. 

Physical  Compatibility.  Physical  compatibility  involves  the 
interconnecting  wires,  cables,  and.  mechanical  linkages.  For 
example,  the  connectors  of  interconnecting  cable  must  fit  the 
outlets  on  the  equipment.  The  general  considerations  of 
physical  compatibility  are  not  addressed  in  this  chapter,  but 
are  included  in  the  area  of  system  performance. 

Man-Machine  Interface  Compatibility.  Compatibility  also 
includes  man-machine  interfaces.  These  interfaces  are  part  of 
MANPRINT  evaluation. 

Mutual  Interference.  Electromagnetic  Compatibility  (EMC)  and 
Electromagnetic  Interference  (EMI)  are  other  aspects  of 
compatibility  which  address  the  extent  to  which  a  system's 
performance  is  degraded  by  its  proximity  to  another  system. 

EMC  and  EMI  are  evaluated  for  their  impact  on  the  electro¬ 
magnetic  transmissions  of  multiple  interfacing  systems  as  well 
as  their  impact  on  friendly  systems  for  which  interfacing  is 
not  intended.  Specifically,  if  two  systems  having  electrical 
transmissions  are  not  integrated  and  are  brought  together  in 
the  field,  then  consideration  of  their  mutual  operation  in  the 
presence  of  each  other  is  addressed  in  EMC.  EMI  addresses 
interference  of  components  within  the  same  system. 


Figure  15-1.  C&I  TERMINOLOGY. 
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Figure  15-2.  Notional  Evaluation  Adequacy  Matrix 
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Chapter  16 

Modeling  and  Simulation  in  Support  of  Test  and  Evaluation 


Section  I 
Overview 


16-1.  Purpose/Scope 

a.  DODM  5000. 2-M,  Part  7  suggests  the  use  of  models  and 
simulations  in  test  and  evaluation  (T&E) .  Modeling  and 
simulation  (M/S)  will  be  considered  to  support  the  developmental 
and  operational  T&E  of  software  and  systems  as  they  proceed 
through  the  life  cycle.  Use  of  M/S  will  include,  but  not  be 
limited  to,  the  identification  of  test  parameters  and  drivers  for 
field  tests;  determination  of  high  risk  areas;  prediction  of  test 
results;  assisting  in  the  allocation  of  scarce  test  resources; 
and  the  assessment  of  system  capabilities  in  situations  which 
cannot  be  tested  due  to  safety,  cost,  or  other  constraints. 
Simulators,  emulators,  drivers,  and  stimulators  (SEDS)  which  are 
used  to  fully  work  load  a  system  under  test  are  also  included. 
Simulators  are  not  to  be  confused  with  threat  simulators  which 
are  covered  in  Part  VIII.  The  extent  of  the  use  of  M/S;  whether 
existing  M/S  will  ever  be  used  or  new  M/S  will  be  developed; 
status  of  M/S  verification,  validation,  and  accreditation  (W&A) ; 
and  the  degree  to  which  M/S  will  augment  test  data  to  assist  in 
system  evaluations  and  assessments  will  be  documented  in  the  Test 
and  Evaluation  Master  Plan  (TEMP) .  Models  and  simulations  used 
for  T&E  must  be  accredited  and  validated  prior  to  their  use  for 
extrapolation  or  predicting  system  performance  (including 
software,  hardware,  man-in-loop).  Software  T&E  procedures  are 
documented  in  Part  Seven. 

b.  This  chapter  provides  background  on  the  use  of  models  and 
simulations,  simulators,  emulators,  drivers,  and  stimulators  to 
support  or  supplement  T&E.  Tt  addresses  requirements  and 
opportunities  for  the  use  of  models  and  simulations  in 
conjunction  with  both  developmental  and  operational  testing.  It 
addresses  a  wide  range  of  modeling  activity  from  engineering 
models  through  item  level  performance  models  to  force  on  force 
models.  Table  16-1  identifies  organizational  points  of  contact 
for  the  various  levels  of  models  to  be  considered  in  test  and 
evaluation  and  simulation  support. 

(INSERT  TABLE  16-1) 
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Section  II 
General 


16-2 .  Background 

a.  Testing  is  conducted  due  to  the  requirement  for  realistic 
and  effective  evaluations  of  Army  system  acquisitions.  Effective 
evaluations  are  required  by  the  decision  maker  to  support 
decision  making  within  the  acquisition  cycle.  T&E  are 
interrelated  and  complementary  processes,  both  of  which  are 
necessary;  however,  neither  process  alone  is  sufficient. 
Evaluations  judge  overall  system  performance  against  technical 
and  operational  mission  requirements  and  reassess  performance  as 
the  mission  requirements  and  system  design  evolve.  The  test 
results,  among  other  sources  of  information,  are  an  integral  part 
of  the  evaluation.  Analytical  models  and  simulations  are  tools 
to  assist  these  processes.  A  consistent  and  traceable  set  of 
tools  should  be  used  throughout  the  T&E  process  to  help  mitigate 
surprises  encountered  during  testing  or  analysis  of  test  results. 
The  framework  for  the  T&E  process,  including  the  use  of  M/S, 
should  be  documented  and  updated  in  the  TEMP. 

b.  M/S  will  continue  to  be  used  extensively  to  support  the 
weapon  development  T&E  process  which  includes  the  software 
development  T&E  process.  Army  systems  in  development  are 
increasingly  complex.  In  addition,  systems  are  required  to 
operate  in  adverse  environments  (weather,  hostile  and  benign 
electromagnetic,  space,  enemy  countermeasures) .  To  be  effective, 
they  must  interact  with  other  systems  often  over  extended  ranges. 
Development  of  advanced  command  and  control  (C&C)  systems  are 
required  to  overcome  these  difficulties.  Furthermore,  C&C 
systems  now  being  developed  are  expected  to  be  effective  against 
future  threats  that  are  not  fully  predictable.  An  extensive  test 
of  such  systems  would  be  large  in  scope  and  require  the 
duplication  of  conditions  that  are  difficult  if  not  impossible  to 
create  short  of  actual  combat.  The  practicalities  of  cost,  test 
range  space,  availability  of  advanced  threat  systems/surrogates, 
safety,  etc.,  will  necessarily  limit  test  planning  and  test  data 
availability.  M/S  provides  an  approach  to  address  these 
limitations.  Evaluation  of  a  major  system's  performance  may 
require  a  "model"  to  integrate  the  available  test  data  and  to 
extrapolate  to  those  conditions,  which  cannot  be  achieved  due  to 
test  constraints  and  limitations  in  the  test  environment.  These 
models  and  simulations  are  not  replacements  for  testing,  but 
rather  complementary  tools  to  assist  in  the  evaluation  process. 

16-3.  Types  of  Models  Used 
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a.  In  the  broadest  sense,  a  model  is  a  representation  of  an 
object,  system,  or  process  (or  a  part  thereof)  in  mathematical, 
physical,  or  logical  terms.  It  is  usually  simplified,  often 
idealized  or  abstract,  and  serves  as  a  basis  for  calculations, 
predictions  or  further  investigations.  Simulation  is  a  technique 
for  experimentation  in  which  the  operation  and  dynamics  of  a 
real-world  system  are  represented  by  the  exercise  of  some 
different  system,  usually  involving  one  or  more  models.  Under 
these  definitions,  a  scaled  representation  of  an  aircraft  is  a 
model;  placing  that  representation  in  a  wind  tunnel  to  study 
flight  dynamics  would  be  a  simulation.  The  basic  concept 
underlying  these  definitions  is  that  a  model  is  an  abstraction 
that  embodies  our  understanding  of  a  system  while  a  simulation 
represents  the  dynamic  exercise  of  one  model  or  a  set  of  models. 

b.  The  Army  utilizes  a  wide  range  of  types  of  models  in  the 
T&E  process:  1.)  engineering  models;  e.g.,  those  that  emulate 
every  steering  command  sent  to  the  control  mechanisms  of  guided 
weapons,  those  that  provide  six-degree  of  freedom  representations 
of  weapon  trajectories,  those  that  address  countermeasures  by 
taking  into  account  propagation  characteristics  of  the  atmosphere 
as  well  as  target  susceptibilities;  2.)  hardware- in-the-loop 
simulations;  e.g.,  those  involving  a  marriage  of  developmental 
hardware  and  software  and  other  equipment  or  stimuli  with  which 
the  developmental  system  must  be  able  to  interact  or  function  on 
the  battlefield;  3.)  battlefield  environment  models;  e.g.,  those 
that  represent  natural  and  man-made  aspects  such  as  smoke,  dust, 
and  obscuration,  those  that  estimate  terrain  effects  on  system 
mobility  characteristics,  those  that  estimate  one-sided 
performance  of  a  system,  those  that  predict  reliability  and  those 
that  address  supportability  issues;  4.)  combat  models;  e.g.,  one 
versus  one  models,  such  as  duels  between  weapons  or  jammers 
against  radars,  force-on-force  models,  ranging  in  force  size  from 
several  elements  on  each  side  to  the  corps  level. 

c.  "Item  level”  models  are  generally  one-sided,  i.e.,  the 
effectiveness  of  a  new  tank  may  be  characterized  by  representing 
its  target  acquisition  and  firepower  capabilities  versus  threat 
tank  attributes  such  as  signature,  size,  speed,  maneuverability, 
and  protection.  The  survivability  of  the  new  tank  might  be 
examined  by  turning  the  one-sided  model  around  and  estimating  the 
threat  tank  capabilities  against  the  new  tank  attributes.  Item 
level  automotive  performance  models  consider  engine  horsepower, 
vehicle  weight,  vehicle  weight  distribution,  traction,  and  other 
attributes  along  with  terrain  characteristics  to  examine 
mobility.  An  item  level  model  can  draw  upon  engineering  models 
for  input,  and  it  can  then  provide  input  to  the  lower  resolution 
models  in  the  hierarchy  described  above  (i.e.,  duels  and  force- 
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on- force) . 

d.  The  analyses  and  evaluations  that  support  the  acquisition 
decision  process  usually  include  the  use  of  force-on-force  models 
that  assist  in  the  evaluation  of  the  synergistic  aspects  of  a 
developmental  system's  contributions  to  total  force 
effectiveness,  [e.g.,  a  new  counterbattery  radar  will  be 
evaluated  in  a  force-on-force  model.  The  model  accounts  for  the 
command,  control,  and  communications  links  to  both  the 
counterfire  weapons,  and  to  other  friendly  elements  that  are 
supported  by  or  protected  by  the  developmental  system.  The  model 
provides  for  equal  representation  of  the  threat  force.  The 
threat  representation  includes  countermeasures  that  can  jam  the 
radar  or  its  communications.  Threat  elements  can  detect  and 
attack  the  radar  and  its  command  and  control.  The  threat  force 
will  include  a  representation  of  its  artillery  capabilities 
versus  the  friendly  force  so  that  the  developmental  radar's 
contributions  can  be  evaluated  in  terms  of  the  survivability  of 
the  total  force] . 

16-4.  Applications  of  Modeling  and  Simulation 

a.  Support  Evaluation/Test  Design  Planning. 

(1)  M/S  can  assist  in  the  T&E  planning  process  and  can 
reduce  the  cost  of  the  conduct  of  testing.  Areas  of  particular 
application  include  scenario  development;  analysis  of  tactics  and 
doctrine  for  systems;  timing  of  test  events;  the  development  of 
objectives,  essential  elements  of  analysis,  and  measures  of 
effectiveness;  the  identification  of  variables  for  control  and 
measurement,  training  test  participants  in  preferred  tactics  and 
doctrine,  and  the  development  of  data  collection,  instrumentation 
and  data  analysis  plans.  For  example;  using  M/S,  the  test 
designer  can  examine  system  sensitivities  to  changes  in  variables 
to  determine  the  critical  variables  and  the  appropriate  ranges  of 
values  to  be  tested.  The  test  designer  can  also  predict,  prior 
to  testing,  the  effects  of  various  assumptions  and  constraints 
and  can  evaluate  candidate  measures  of  effectiveness  to  help  in 
formulation  of  the  test  design  plan. 

(2)  Caution  must  be  exercised  when  planning  to  rely  on  M/S 
as  a  means  of  obtaining  test  data.  M/S  tend  to  be  very  expensive 
to  develop.  M/S  results  are  difficult  to  integrate  with  data 
from  other  sources,  and  often  do  not  provide  the  level  of  realism 
required  from  testing.  Although  M/S  have  these  limitations 
compared  to  testing,  they  should  be  used,  whenever  appropriate, 
as  another  source  of  data  for  the  evaluator. 
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(3)  Computer  simulations  may  be  used  to  test  the  planning 
for  a  test  exercise.  By  setting  up  and  running  the  planned  test 
exercise  in  a  simulation,  the  timing  of  events  and  scenario  may 
be  tested  and  validated.  The  interaction  of  the  various  forces 
which  test;  the  measures  of  effectiveness,  the  essential  elements 
of  analysis  and,  in  turn,  the  test  objectives  may  identify 
critical  events.  Further,  the  simulation  may  be  used  to  verify 
the  statistical  test  design,  the  instrumentation  plan,  the  data 
collection  plan,  and  the  data  analysis  plan.  Essentially,  ^he 
purpose  of  the  simulation  in  pretest  planning  is  to  preview  the 
test  exercise  to  make  test  planning  more  efficient. ^  Pretest  ^ 
planning  attempts  to  optimize  test  conduct  by  avoiding  potential 
trouble  spots  and  increasing  the  potential  for  efficient  data 
collection. 

(4)  As  an  example  of  a  simulation  used  in  test  planning, 
consider  a  model  which  portrays  aircraft  versus  air  defenses. 

The  model  can  be  used  to  replicate  typical  scenarios  and  provide 
data  on  the  number  of  engagements,  the  air  defense  systems 
involved,  the  aircraft  target,  the  length  and  quality  of  the 
engagement  and  a  rough  approximation  of  the  success  of  the 
mission  (i.e.,  did  the  aircraft  make  it  to  the  target?).  With 
such  information  available,  a  data  collection  plan  can  be  ,  ,  . 
developed  to  specify  in  more  detail  when  and  where  data  should  be 
collected,  from  which  systems,  and  in  what  quantity.  The  results 
of  this  analysis  can  be  extremely  useful  in  planning  for  long 
lead  time  items  such  as  data  collection  devices  and  data 
processing  systems.  As  tactics  are  developed  and  typical  flight 
paths  are  generated  for  the  scenario,  an  analysis  can  be 
performed  on  the  flight  paths  over  the  terrain  in  <^estion.  A 
determination  can  be  made  whether  or  not  the  existing 
instrumentation  can  track  the  numbers  of  aircraft  involved  in 
their  maneuvering  envelopes.  Trade-offs  can  be  made  between  the 
amount  of  equipment  to  be  purchased  and  the  types  of  profiles 
which  can  be  tracked  for  this  particular  test.  Use  of  such  a 
model  can  also  highlight  numerous  choices  available  to  the  threat 
air  defense  system  in  terms  of  opportunities  for  engagement,  and 
practical  applications  of  doctrine  to  the  specific  situations. 


b.  Support  Test  Execution. 

(1)  Simulations  can  be  useful  in  test  execution  and 
dynamic  planning.  With  funds  and  other  restrictions  limiting  the 
number  of  times  that  a  test  may  be  repeated  and  with  tests 
sometimes  requiring  several  days  for  completion,  it  is  mandatory 
that  the  test  director  exercise  close  control  over  conduct  of  the 
test  elements.  This  is  to  ensure  that  planned  data  are  being 
gathered  and  to  ensure  adequate  safety.  The  test  director  must 
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be  able  to  make  minor  modifications  to  the  test  plan  and  scenario 
to  adapt  the  test  activities  to  changing  circumstances.  This 
calls  for  a  quick-look  analysis  capability  and  a  dynamic  planning 
capability.  Simulations  may  contribute  to  this  capability,  e.g., 
using  the  same  simulation (s)  as  used  in  the  pretest  planning,  the 
tester  could  input  data  gathered  during  the  first  day  of  the 
exercise  to  determine  the  adequacy  of  the  data  to  fulfill  the 
test  objectives.  Using  these  data,  the  entire  test  could  be 
simulated  to  isolate  and  to  modify  the  test  plans  to  account  for 
these  projected  inadequacies.  Portions  of  the  test  could  be 
deleted  to  save  resources. 

(2)  Simulations  may  also  be  used  to  support  test  control 
and  to  assure  safety,  e.g.,  during  missile  test  firings, 
aerodynamic  simulations  of  the  proposed  test  were  run  on  a 
computer  during  actual  firings  so  that  real  time  missile 
position  data  could  be  continuously  compared  to  the  simulated 
missile  position  data.  For  example,  if  variations  occurred 
Beyond  acceptable  limits  and  if  the  range  safety  officer 
relinquished  manual  control,  the  computer  could  issue  a  command 
to  destruct  the  missile. 

(3)  Simulations  can  augment  test  execution  in  order  to 
reduce  costs.  For  example,  in  air  defense  systems,  missile  fly¬ 
out  simulations  may  be  used  in  conjunction  with  system  testing  to 
reduce  the  expenditure  of  live  missiles  while  providing 
information  on  overall  system  performance.  Simulations  can  also 
be  used  to  augment  test  execution  by  providing  a  means  to 
simulate  stress  (or  workload)  to  the  system  under  test.  For 
example,  it  may  be  prohibitive  to  set  up  a  network  of  message 
traffic  to  adequately  stress  a  communication  system,  but  a 
simulation  may  be  used  to  provide  sufficient  (simulated)  message 
traffic  to  the  system  under  test  at  a  reasonable  cost. 

(4)  Simulations  can  be  used  to  augment  tests  by  simulating 
non-testable  events  and  scenarios.  Although  testing  should  be 
accomplished  in  as  realistic  an  environment  as  possible, 
pragmatically,  some  environments  are  impossible  to  simulate  for 
safety  or  other  reasons.  Some  of  these  include  the  environment 
of  a  nuclear  battlefield  to  include  the  effects  of  nuclear  bursts 
on  both  friendly  and  enemy  elements.  Others  include  live  firings 
of  opposing  forces  and  adequate  representation  of  other  forces  to 
obtain  compatibility  and  interoperability  data.  Instrumentation, 
data  collection,  and  data  reduction  of  large  combined  arms  forces 
(e.g.,  brigade,  division,  and  larger-sized  forces)  become 
extremely  difficult  to  control  and  costly  to  execute. 

(5)  Usually,  insufficient  units  are  available  to  simulate 
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the  organizational  relationships  and  interaction  of  the  equipment 
with  its  operational  environment,  particularly  during  the  early 
OT&E  conducted  using  prototype  or  pilot  production  type 
equipment.  Simulations  are  not  constrained  by  these  limitations. 
Data  obtained  from  a  limited  test  can  be  used  in  more  complex 
simulations  to  model  scenarios  involving  the  interaction  of 
friendly  forces  against  a  variety  of  threat  systems. 

(6)  Simulations  can  also  play  design  characteristics  of 
equipment  and  can  be  used  to  augment  the  results  obtained  using 
prototype  equipment  that  is  "mocked-up”  to  represent  the  final 
item.  The  simulation  may  be  used  to  represent  production  level 
equipment  in  those  areas  where  the  prototype  cannot  meet 
production  level  performance. 

(7)  It  is  often  necessary  to  use  "substitute”  systems  in 
testing,  e.g.,  an  on-hand  available  system  is  used  to  represent 
the  required  system.  The  "substitute"  system  may  have  greater  or 
less  capabilities  than  the  desired  system.  Simulations  are 
capable  of  representing  the  actual  characteristics  of  systems 
and,  therefore,  can  be  used  as  a  means  of  modifying  raw  data 
collected  during  the  test  to  estimate  the  required  system 
characteristics.  As  an  example,  suppose  the  substitute  system  is 
an  anti-aircraft  artillery  gun  with  a  tracking  rate  of  30  degrees 
per  second.  The  required  system  has  a  tracking  rate  of  45 
degrees  per  second.  A  computer  simulation  could  be  used  to 
augment  the  collected  test  data  by  estimating  the  number  of 
rounds  which  would  have  been  fired  against  each  target  or  whether 
targets  that  were  missed  because  of  the  slower  tracking  rate 
could  have  been  engaged  by  the  required  system. 

c.  Support  Analysis. 

(1)  Modeling  and  simulation  may  be  used  in  post-test 
analysis  to  extend  and  generalize  results  and  to  extrapolate  to 
other  conditions.  The  cost  and  difficulty  of  controlling  large 
test  exercises,  not  to  mention  the  difficulty  in  instrumenting 
them  and  collecting  and  reducing  the  data,  to  some  degree  limits 
the  size  of  OT&E.  This  makes  the  process  of  determining  the 
operational  suitability  of  equipment  to  include  compatibility, 
interoperability,  organization,  etc. ,  a  difficult  one.  To  a 
large  degree,  the  interactions,  interrelationships,  and 
compatibility  of  large  forces  may  be  obtained  by  using  data 
collected  during  the  test  and  playing  it  in  a  simulation. 

(2)  Simulations  can  be  used  to  extend  test  results,  save 
considerable  test  resources,  and  reduce  test  cost.  This  is 
accomplished  by  minimizing  the  need  to  replicate  tests  for  the 
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improvement  of  statistical  sample,  determining  overlooked  or 
directly  unmeasured  parameters. 

(3)  During  analyses  the  test  results  can  be  compared  to 
the  results  predicted  by  the  simulations  used  early  in  the 
planning  process.  Thus,  the  simulation  may  be  validated  by  the 
actual  live  test  exercise  results,  and  the  test  exercise  may  gain 
credibility  from  the  comparison  with  the  simulation. 

16-5.  M/S  in  the  Test  and  Evaluation  Master  Plan  (TEMP) 

a.  During  the  test  and  evaluation  planning  process,  both  the 
tester  and  the  evaluator  determine  to  what  degree  M/S  are 
necessary  to  supplement  the  T&E  process  and  they  forecast 
requirements  for  the  use  of  M/S  to  support  T&E  of  the  system 
during  the  system's  life  cycle.  Although  some  evaluations  such 
as  confirmation  of  critical  technical  and  operational  parameters 
are  performed  in  a  traditional  testing  environment,  others  such 
as  safety,  excursions,  logistics  support  are  conducted  in  a 
simulated  or  modeled  environment.  Development  of  the  system's 
evaluation  plan  is  often  based  on  a  trade-off  analysis  that 
identifies  the  appropriate,  affordable  evaluation  tools  (M/S 
and/or  testing)  to  address  the  acquisition  decision  issues. 
Additionally,  the  evaluation  plan  should  establish  the 
complementary  roles  that  M/S  and  testing  will  play  to  support  the 
analytic  process.  The  types  of  M/S,  their  applications,  and 
their  resource  requirements  are  documented  in  the  TEMP. 

b.  The  TEMP  defines  and  integrates  the  DT&E  and  OT&E  efforts 
for  all  major  weapon  system  procurements  and  related  software 
development  efforts.  It  relates  program  schedule,  decision 
milestones,  test  management  structure,  and  test  resources  to 
critical  technical  parameters,  COIC,  additional  operational 
issues  and  criteria  (AOIC) ,  and  T&E  procedures.  It  is  used  as  a 
tool  for  oversight,  review,  and  approval  of  the  test  and 
evaluation  effort  by  Office  Secretary  of  Defense  (OSD)  and  all 
Department  of  Defense  (DoD)  components.  The  TEMP,  as  described 
in  DODM  5000. 2-M,  explicitly  covers  the  system  requirements, 
program  summary,  DT&E,  OT&E,  and  test  and  evaluation  process. 

16-6.  The  Concept  of  Linkage 

a.  Decisions  concerning  the. use  of  M/S  should  be  made  as 
early  in  the  acquisition  cycle  as  possible  to  support  the  timely 
development  of  any  new  modeling  or  any  required  upgrades  to 
existing  models.  This  requires  early  coordination  (as  soon  as 
possible  after  Milestone  0) among  the  user,  developer,  testers  and 
evaluators.  Ideally,  they  will  agree  not  later  than  Milestone  I, 
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on  the  M/S  needed  to  provide  assessments  for  the  systems.  A 
plan  should  be  developed  at  this  time  to  transition  the  generic 
modeling  capability  specified  at  Milestone  I  to  a  mature,  fully 
defined  M/S  requirement  by  Milestone  II  (for  execution  between 
Milestones  II  and  III).  In  addition,  data  requirements  and 
acquisition  of  input  data  for  models  are  critical  issues  and  are 
often  long  lead  time  items. 

b.  To  ensure  the  validity  of  analyses  used  to  evaluate 
capabilities  in  the  context  of  complete  force  interactions  and 
varying  battlefield  conditions,  the  models  and  computerized 
simulations  upon  which  these  analyses  are  based  must  correctly 
represent  threat  force  system  capabilities,  threat  force  combat, 
threat  combat  support,  and  threat  combat  service  support. 

Although  treatment  of  the  threat  is  not  specifically  addressed  in 
the  model  architecture,  threat  is  generally  addressed  through 
equal  representation  in  the  model  input  data,  decision  rules,  and 
scenario  specifications  (i.e.,  force  structure  and  composition, 
weapons/munitions) .  Those  conducting  analyses  must  know  the 
strength,  weaknesses,  and  limitations  of  any  model  selected  with 
respect  to  representation  of  the  threat.  Threat  data  provided  by 
supporting  intelligence  organizations  must  be  accurate,  current, 
and  derived  from  validated  intelligence  products  and  data  bases. 
lAW  AR  381-11,  deviations  from  validated  scenarios  or  threat 
data,  must  be  documented  and  forwarded  through  the  Operational 
Test  and  Evaluation  Command  (OPTEC) ,  and  the  Deputy  Chief  of 
Staff  for  Operations  and  Plans  (DCSOPS)  to  the  Department  of  the 
Army  Deputy  Chief  of  Staff  for  Intelligence  (DA  DCSINT)  for 
approval.  Results  of  the  analyses  must  indicate  any  deviations 
and  associated  implications. 

c.  The  TEMP  is  the  principle  document  for  test  scheduling  and 
planning  for  the  T&E  community  and  program  manager.  The  TEMP 
should  document  required  analyses  and  testing.  COEA  analyses,  DT 
results  and  OT  results  should  be  synthesized  and  analyzed  to 
provide  a  comprehensive  evaluation. 

d.  Current  acquisition  policy  (DOD  5000  series)  states  that 
the  COEA  and  test  and  evaluation  are  aids  to  decision-making. 

The  COEA  aids  decisionmakers  in  judging  whether  any  of  the 
proposed  alternatives  offers  a  cost  effective  approach  to  meeting 
the  operational  requirement.  Test  and  evaluation  aids 
decisionmakers  by  verifying  that  systems  have  attained  their 
technical  performance  specifications  and  objectives  and  that  they 
are  operationally  effective  and  operationally  suitable  for  their 
intended  use. 

e.  Current  acquisition  policies  (DOD  5000  series)  also  state 
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that  a  linkage  should  exist  between  COEAs  and  test  and 
evaluation.  Specifically,  DODI  5000.2,  Part  4,  Section  E, 
paragraph  3. a (5)  Measures  of  Effectiveness  states: 

"To  judge  whether  an  alternative  is  worthwhile,  one  must  first 
determine  what  it  takes  to  make  a  difference.  Measures  of 
effectiveness  should  be  defined  to  measure  operational 
capabilities  in  terms  of  engagement  or  battle  outcomes. 
Measures  of  performance,  such  as  weight  and  speed,  should 
relate  to  the  measures  of  effectiveness  such  that  the  effect 
of  a  change  in  the  measure  of  performance  can  be  related  to  a 
change  in  the  measure  of  effectiveness. 

(a)  Comparable  measures  for  each  alternative  are 
evaluated  against  a  baseline,  generally  the  outcome  that  would 
exist  with  currently  programmed  capabilities. 

(b)  The  complexity,  scope,  and  output  measures  of 
mathematical  models  selected  for  the  analysis  should  be 
appropriate  to  the  system  being  evaluated.  For  example,  a 
battalion  size  model  need  not  be  run  to  evaluate  a  new  truck,  and 
an  antisubmarine  warfare  campaign  model  is  not  necessary  for 
assessing  the  performance  of  new  carrier  onboard  delivery 
systems . 


(c)  Measures  of  effectiveness  should  be  developed  to  a 
level  of  specificity  such  that  a  system's  effectiveness  during 
developmental  and  operational  testing  can  be  assessed  with  the 
same  effectiveness  criteria  as  used  in  the  cost  and  operational 
effectiveness  analysis.  This  will  permit  further  refinement  of 
the  analysis  to  reassess  cost  effectiveness  compared  to 
alternatives  in  the  event  that  performance,  as  determined  during 
testing,  indicates  a  significant  drop  in  effectiveness  (i.e.,  to 
or  below  a  threshold)  compared  to  the  levels  assumed  in  the 
initial  analysis." 

DODM  5000. 2-M,  Part  8,  paragraph  2.a(5)  states: 

"A  comprehensive  test  and  evaluation  program  is  an  integral 
factor  in  analyzing  operational  effectiveness,  since  it 
will  provide  test  results  at  each  milestone  decision  point 
that  give  credence  to  the  key  assumptions  and  estimates 
that  may  have  been  made  in  the  current  or  earlier  cost  and 
operational  effectiveness  analyses". 


16-7.  Credibility  of  Models  and  Simulations 
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a.  As  the  use  of  models  and  simulations  has  increased,  the 
models  and  simulations  themselves  have  grown  in  size  and 
complexity.  In  fact,  many  simulations  that  support  the 
acquisition  process  are  too  complicated  to  be  sufficiently 
understandable  by  decision-makers,  testers,  and  analysts  in  a 
cursory  review.  This  has  led  to  concerns  related  to  their 
application.  It  is  important  to  ensure  that  the  results  of 
simulations  used  to  support  major  test  and  evaluation  are 
credible.  Steps  must  be  taken  to  assure  a  high  level  of 
confidence  in  the  results  provided  by  such  models  and  simulations 
and  that  the  proper  model  has  been  used  to  address  the 
acquisition  decision  issues. 

b.  An  essential  attribute  of  any  useful  M/S  is  that  it  has 
earned  a  high  degree  of  credibility;  i.e.,  its  construction, 
execution,  and  the  interpretation  of  its  output  results  are 
considered  to  be  "good  and  true",  when  taken  in  the  proper 
context  by  a  community  of  peers. 

c.  System  and  subsystem  models  at  the  engineering  level, 
particularly  those  which  model  functions  that  do  not  represent 
combat  environments,  seem  to  enjoy  a  fairly  high  level  of 
credibility  among  analysts  and  decision  makers,  e.g.,  it  is 
virtually  impossible  to  imagine  a  modern  aircraft  development 
program  that  would  not  make  extensive  use  of  wind  tunnels  and 
flight  dynamics  simulations.  It  has  been  shown  that  these 
simulations  frequently  permit  a  reduction  in  the  number  of 
aircraft  flight  hours  required  during  the  development  process. 
This  high  level  of  simulation  credibility  can  be  attributed  to 
the  degree  of  development  of  aerodynamic  sciences  (at  least 
empirically)  and  the  use  of  instrumented  aircraft  flight  test 
data  collected  to  continually  improve  the  fidelity  of  such 
simulations. 

d.  Complex  combat  M/S  which  estimate  operational  performance 
naturally  encounter  more  skepticism  among  decision  makers  due  to 
the  M/S  simplifying  assumptions  and  the  fact  that  the  fundamental 
theoretical  mathematical  bases  for  aggregation  and  disaggregation 
are  less  well  understood.  The  difficulty  in  representing  the 
impact  of  human  performance  factors  in  combat  (stress,  fatigue, 
shock,  etc.)  adds  to  the  skepticism.  It  could  be  argued  that 
only  actual  combat,  with  instrumentation  to  collect  data,  could 
fully  resolve  all  the  suspect  elements  of  simulation.  The  use  of 
multiple  models  with  different  theoretical  approaches  and 
assumptions  may  provide  a  hedge  against  the  uncertainty  of  our 
fundamental  knowledge  of  combat  processes  and  of  our  ability  to 
implement  these  processes  into  a  model.  These  different 
approaches  and  assumptions  must  be  made  clear,  however,  or  the 
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different  models  are  likely  to  generate  more  confusion  than 
insight. 

e.  In  order  to  promote  credibility  of  M/S,  HQDA  issued  a 
policy  memorandum  in  October  1989  which  mandated  the  validation, 
verification  and  accreditation  (W&A)  of  models  and  simulations 
used  as  acquisition  decision  making  tools  in  the  Army  acquisition 
process.  In  June  1990,  an  additional  Army  policy  was  issued 
which  extended  the  W&A  mandate  to  include  research,  development 
and  engineering  (RD&E)  models  and  simulations.  Both  Army 
policies  have  been  incorporated  into  AR  5-11,  Army  Model  and 
Simulation  Management  Program  dated  10  June  1992.  Samples  of 
credibility  issues  that  need  to  be  addressed  when  applying  models 
and  simulations  to  test  and  evaluation  are  contained  in  Table  16- 
2.  Prior  to  using  any  M/S  (where  the  results  will  be  used  in  the 
Army  acquisition  decision  making  process) ,  it  must  undergo  a 
formal  W&A.  Information  on  models  which  have  been  formally 
accredited  can  be  obtained  from  the  Army  Model  Improvement  and 
Study  Management  Agency. 

(INSERT  TABLE  16-2) 


Section  III 

Examples  of  Using  M/S  in  Developmental  Test  and  Evaluation  (DT&E) 


16-8.  General 

a.  Developmental  testing  is  a  generic  term  which  encompasses 
engineering-type  testing,  and  software  development  testing.  It 
addresses  the  systems  technical  characteristics  and  contributes 
to  the  acquisition  and  fielding  of  an  effective,  supportable,  and 
safe  system.  Generally,  it  requires  instrumentation  and 
measurements,  and  is  accomplished  by  engineers,  technicians  or 
soldier  operator-maintainor  test  personnel.  The  categories  of 
developmental  test  are  defined  in  Chapter  4,  AR  73-1.  M/S 
opportunities  to  supplement  these  tests  are  numerous.  The 
following  examples,  while  not  exhaustive,  represent  instances 
where  M/S  have  been  used  effectively  in  developmental  T&E.  In 
addition,  examples  of  the  application  of  M/S  in  T&E  are  listed  in 
Table  16-3. 


(INSERT  TABLE  16-3) 

16-9.  PATRIOT  Air  Defense  System 

a.  Extensive  use  of  modeling  and  simulation  has  been  made 
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during  the  development,  testing,  fielding,  and  upgrading  of  the 
PATRIOT  Air  Defense  System.  PATRIOT  simulations  include: 

(1)  The  Flight  Mission  Simulator  (FMS) .  The  FMS  is  a 
device  which  loads  the  PATRIOT  radar  by  supplying  inputs 
representing  the  electronic  countermeasure  (ECM)  environment, 
natural  clutter,  multiple  maneuvering  targets,  and  missiles  in 
flight.  The  PATRIOT  fire  unit  can  conduct  normal  surveillance 
and  engagements  concurrently  with  FMS  loading.  To  exercise  the 
PATRIOT  system  to  its  full  specified  capability,  simulated 
missile  firings  are  conducted  against  simulated  targets  with 
target  loading  provided  by  a  combination  of  actual  aircraft  and 
aircraft  simulated  by  the  FMS. 

(2)  The  Guidance  Test  Support  Facility  (GTSF) .  The  GTSF 
is  a  hardware-in-the-loop  simulation  of  the  PATRIOT  guidance 
system.  It  provides  a  detailed  simulation  of  the  guidance  system 
and  missile  response  for  one  missile  flyout.  This  simulation  is 
used  for  missile  preflight  test  performance  prediction,  post- 
flight  analysis,  system  performance  evaluation,  and  ECM  analysis. 

(3)  H-1.  The  H-1  is  a  detailed  hybrid  simulation  of  the 
PATRIOT  guidance  system  from  prelaunch  through  intercept.  This 
simulation  is  used  for  preflight  and  postflight  analysis.  The 
guidance  and  control  characteristics  that  exist  as  actual 
hardware  in  the  GTSF  are  simulated  in  H-1. 

(4)  S-1.  The  S-1  is  a  Monte  Carlo  digital  simulation  of 
the  PATRIOT  radar  and  Engagement  Control  Station  (ECS)  resident 
software.  The  outputs  of  S-1  include  radar  surveillance  and 
kinematic  intercept  ranges  which  provide  a  basis  for  estimating 
the  probability  of  detection,  evaluation,  and  transfer. 

(5)  Lethality  Endgame  Simulation  (LEGS).  LEGS  is  a 
digital  endgame  simulation  that  uses  Monte  Carlo  techniques  to 
calculate  probability  of  kill.  LEGS  includes  an  endgame 
kinematics  model,  a  fuze  model,  a  warhead  model,  and  a  lethality 
prediction  model.  Intercept  geometries  are  defined  by  flight 
test  data  or  H-1  simulation  results.  LEGS  is  used  to  evaluate 
the  probability  of  kill  against  threat  targets  based  upon  flight 
test  intercept  geometry. 

b.  Use  of  these  simulations  has  allowed  more  complete 
evaluations  of  the  PATRIOT  system  with  fewer  required  flight 
tests.  Some  specific  examples  follow: 

(1)  Prototype  Qualification  Test  -  Government  (PQT- 
G) /Engineering  and  Manufacturing  Development  (EMD) .  Use  of  the 
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Flight  Mission  Simulator  permitted  the  specified  target  loading 
and  multiple  simultaneous  engagement  capability  to  be  tested.  It 
allowed  a  large  number  of  targets/engagements  to  be  simulated 
while  the  radar  was  emitting  and  processing  returns  from  terrain, 
chaff,  and  live  Standoff  Jammer  (SOJ) ,  Escort  Jammer  (ESJ) ,  and 
Self  Screening  Jammer  (SSF) .  Without  the  FMS,  the  full 
target/engagement  load  could  not  have  been  tested.  Also,  the 
multiple  simultaneous  engagement  capability  could  be  repetitively 
tested.  An  actual  test  would  have  required  multiple  intercepts 
within  a  very  short  period  of  time,  conditions  that  are  very 
expensive  to  test  with  low  probability  of  success.  The  GTSF  was 
used  during  development  to  help  design  and  improve  the  system, 
and  to  predict  performance.  The  GTSF  allowed  testing  missile 
guidance  logic  for  a  wide  variety  of  ECM  waveforms.  Use  of  the 
GTSF  allowed  repetition  of  ECM  conditions  that  would  not  have 
been  permitted  on  the  test  range.  Although  limitations  on  the 
number  of  signal  channels  did  require  certain  flight  tests  of 
selected  formations  of  SSJ  with  responsive  waveforms  in 
chaff/ground  clutter,  missile  flights  were  used  essentially  to 
confirm  GTSF  results.  Use  of  the  GTSF  substantially  reduced  the 
number  of  flight  tests  and  minimized  using  actual  missiles  to 
find  ECM  limitations  of  the  system. 

(2)  First  Article  Test.  Due  to  the  use  of  the  GTSF,  four 
missile  firings  were  adequate  for  the  technical  evaluation  of 
this  phase,  and  an  additional  four  firings  for  operational 
testing.  The  FMS  was  used  to  verify  both  system  maximum  loading 
capability  and  multiple  simultaneous  engagement  performance.  In 
addition,  a  special  version  of  the  tactical  software  was 
developed  which  simulated  missile  engagements  (did  not  simulate 
radar  and  computer  engagement  loading  as  did  the  FMS) .  This 
allowed  for  a  more  realistic  follow-on  evaluatioh  and  for 
technical  and  operational  tests  to  be  conducted  in  an  environment 
containing  chaff,  SOF's,  SSJ's,  and  numerous  simulated  aircraft. 
More  recently,  simulated  tactical  ballistic  missiles  (TBM's)  have 
been  included  in  the  test  scenarios. 

16-10.  STINGRAY  Countermine  System 

a.  Another  example  of  the  use  of  models  and  simulation  to 
augment  developmental  test  and  evaluation  involves  the  use  of  the 
Low  Energy  Laser  Weapon  Simulation  (LELAWS)  in  the  T&E  of  the 
STINGRAY  system.  STINGRAY  is  intended  to  act  as  a  countermeasure 
to  optical  and  electro-optical  targeting  systems.  LELAWS  is  a 
simulation  which  estimates  the  probability  that  a  given 
countermeasure  was  successful.  In  preparation  for  the  STINGRAY 
Concept  Demonstrator  (CD)  developmental  test,  LELAWS  was  used  to 
estimate  downrange  laser  energy.  This  information  was  used  to 
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determine  STINGRAY  CD  test  instrumentation/calibration 
requirements.  As  LELAWS  was  to  be  used  in  STINGRAY'S  upcoming 
COEA,  the  PM-STINGRAY  authorized  a  number  of  test  runs  dedicated 
solely  to  supporting  the  validation  of  the  LELAWS  methodology. 
After  the  STINGRAY  CD  testing  was  completed  and  the  test  data 
applied  to  LELAWS  validation,  LELAWS  was  used  to  conduct  a  risk 
assessment  on  the  STINGRAY'S  ability  to  meet  user  performance 
requirements  in  its  next  phase  of  development.  The  risk 
assessment  was  conducted  by  inputting  near-term  STINGRAY 
technology  options  into  the  LELAWS  model,  estimating  STINGRAY 
performance  using  the  validated  LELAWS  methodology,  and  comparing 
LELAWS  output  to  STINGRAY  user  requirements. 

16-11.  Mine  Systems 

a.  MINEMIX  has  and  is  being  used  to  supplement  test  and 
evaluation  efforts  related  to  the  development  and  fielding  of  the 
Army's  mine/barrier  systems.  The  MINEMIX  model  is  a  Monte  Carlo 
simulation  structured  to  provide  estimates  of  minefield 
effectiveness  for  various  'mixes'  of  both  anti-personnel  and 
anti-materiel  mines.  Conventional,  scatterable,  and  SMART  mines 
can  be  included  in  the  layout  of  various  minefields.  The 
performance  of  individual  nines  is  described  by  a  collection  of 
parameters  that  indicate:  arming  reliability,  fuzing 
reliability,  target  vulnerability,  probability  of  being  a  'dud', 
countermeasure  probability,  etc.  The  model  provides  the 
capability  of  estimating  the  effectiveness  of  'minefields' 
against  target  arrays  which  are  constructed  of  different  types 
and  numbers  of  vehicles,  e.g. ,  main  battle  tanks,  APCs,  squad 
formations  of  personnel,  and  wheeled  vehicles.  The  matrix  of 
output  produced  by  the  model  enables  the  user  to  depict  the  exact 
configuration  of  the  minefield,  along  with  a  tally  of  the 
"killer/victims",  i.e.,  which  mines  defeated  which  targets. 

b.  In  the  context  of  T&E,  MINEMIX  has  been  used  to  supplement 
limited  testing  of  the  expensive  Wide  Area  Mine  (WAM) .  The 
effectiveness  of  the  WAM  is  a  function  of  the  distance  at  which 
the  mine  detects  and  attempts  to  engage  a  valid  target  that 
enters  its  zone  of  authority.  In  order  to  address  questions  of 
minefield  effectiveness,  MINEMIX  supplements  the  test  data 
acquired  at  specific  'ranges'  and  allows  the  analyst  to  examine 
issues  such  as  the  requirement  for  range  sensitivity,  mine 
spacing,  mine  mixing  with  scatterable  mines,  etc.,  without 
additional  test  results.  The  output  of  the  MINEMIX  simulation 
was  used  to  assist  in  the  formulation  of  valid  requirement 
parameters  that  became  part  of  the  approved  requirements  document 
for  the  Wide  Area  Mine. 
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c.  In  terms  of  reducing  testing,  MINEMIX,  coupled  with  other 
•engineering'  simulations  can  be  used  to  fill  voids  left  by  the 
limited  testing  that  is  planned  for  the  Wide  Area  Mine.  For 
example,  the  effectiveness  of  WAM  is  also  a  function  of  the  point 
on  the  target  which  is  hit  by  the  explosively  formed  penetrator. 
Captive  flight  tests  of  the  sensor  are  used  to  build  a  database 
of  'hit  points'  for  various  launch  conditions  which  then  serve  as 
an  input  array  to  the  MINEMIX  model  to  allow  sensitivity  studies 
concerning  this  key  parameter.  In  addition  to  captive  flight 
tests,  various  engineering  simulations  of  the  sensor/target 
encounter  are  also  used  to  provide  input  to  the  MINEMIX  model  and 
to  supplement  'live  mine'  testing  against  actual  or  surrogate 
targets . 

16-12.  C3I  Systems 

a.  The  Simulation/Stiroulation  (SIM/STIM)  software  is  a  real¬ 
time,  multitasking  system  which  provides  two  functions  related  to 
test  and  evaluation  for  the  Army  Field  Artillery  Tactical  Data 
System  (AFATDS)  testbed.  First,  it  simulates  those  nodes  (units) 
in  the  Brigade  slice  which  are  not  present  within  the  given 
AFATDS  system  under  test  (SUT) .  This  is  accomplished  by  first 
simulating  messages  which  occur  between  simulated  nodes  and 
performing  the  actions  of  the  simulated  nodes  with  realistic 
timing.  Second,  SIM/STIM  simulates  the  real  equipment  within  the 
SUT  by  transmitting  representative  messages  from  the  simulated 
nodes,  as  well  as  receiving  and  processing  message  traffic  from 
the  SUT. 

b.  SIM/STIM  was  first  developed  for  use  in  AFATDS  concept 
evaluation  testing.  Currently,  SIM/STIM  and  its  derivatives  are 
being  planned  as  an  integral  part  of  the  Fire  Support  Automated 
Test  System  (FSATS)  which  is  a  suite  of  test  instrumentation  to 
be  used  to  support  test  planning,  test  design,  test  execution, 
data  reduction,  data  analysis,  and  test  reporting  associated  with 
the  technical  and  operational  tests  of  the  Army's  fire  support 
systems.  Initially,  FSATS  is  envisioned  for  use  in  the  Post 
Production  Test  and  Evaluation  and  in  the  Initial  Operational 
Test  and  Evaluation  of  the  AFATDS  system. 

16-13.  Live  Fire 

a.  A  Live  Fire  Test  is  a  type  of  developmental  test,  the 
objective  of  which  is  to  support  a  timely  and  thorough  assessment 
of  the  vulnerability/lethality  of  a  system  as  it  progresses 
through  its  development  and  subsequent  production  phases.  Live 
Fire  Test  and  Evaluation  (LFT&E)  should  demonstrate  the  ability 
of  the  weapon  system  or  munition  to  provide  for  system 
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suirvivability  or  munition  lethality.  The  objective  of  LFT&E  is 
to  provide  insights  into  the  principal  damage  mechanisms  and 
failure  modes  occurring  as  a  result  of  the  munition/target 
interaction  leading  to  improved  techniques  for  reducing  personnel 
casualties  or  for  enhancing  system  survivability  or  munition 
lethality.  Data  from  LFT&E  should  help  identify  specific  failure 
and  damage  mechanisms.  With  this  knowledge,  cost  effectiveness 
trade-offs  can  be  conducted  to  predict  the  optimal  "mix"  of 
system  vulnerability  reduction/munition  lethality  enhancement 
measures . 

b.  Live  Fire  Testing,  even  when  supplemented  with  other 
developmental  testing,  cannot  produce  sufficient  data  to  assess 
the  vulnerability  or  lethality  of  a  system  for  all  combinations 
of  threat,  impact,  and  engagement  conditions  of  interest.  Thus, 
modeling  must  be  used  to  extend  test  results  to  account  for  the 
conditions  which  are  of  interest  but  are  impractical  or 
impossible  to  test.  In  the  context  of  LFT&E  system, 
vulnerability/munition  lethality  modeling  has  four  basic  roles: 
support  test  design,  support  the  evaluation  of  system  and  crew 
vulnerability  or  munition  lethality,  guide  and  evaluate 
vulnerability  reduction  or  lethality  enhancement  efforts,  and 
methodology  diagnosis. 

c.  Detail  on  the  classes  and  roles  of  system  vulnerability/ 
munition  lethality  modeling  in  LFT&E  are  contained  in  Part  VI. 


Section  IV 

Areas  for  Using  M/S  in  Operational  Test  and  Evaluation  (OT&E) 


16-14.  General 

a.  It  is  impossible  to  address  all  the  possible  or  even 
likely  uses  of  M/S  within  the  OT&E  process.  Categories  of  OT  are 
listed  in  Chapter  5,  AR  73-1.  Test  requirements  span  a  wide 
spectrum  of  conditions  and  are  seldom  standardized.  There  are, 
however,  four  areas  in  which  the  analytical  community  and  M/S  can 
significantly  aid  the  operational  tester:  (1)  Development  (crew 
training  analyses/ evaluations  and  tactics  and  operational 
techniques  development) ;  (2)  Test  planning  and  operational 
scenario  development;  (3)  Data  extrapolation/interpolation  beyond 
test  results;  and  (4)  Test  execution.  These  applications  of  M/S 
can  supplement  testing  and  provide  a  much  more  robust  and 
informative  evaluation.  The  following  concepts  and  examples  are 
intended  to  characterize  basic  principles  and  processes. 
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16-15.  Crew  Training  Analyses/Evaluations 

a.  Operational  testers  have  become  involved  in  training  and 
training  development  through  Force  Development  Test  and 
Evaluations  (FDT&E) .  The  implementation  of  the  FDT&E  process 
stemmed  from  an  identified  need  for  more  comprehensive  training 
of  units  prior  to  operational  testing.  Insights  gained  from 
several  operational  tests  (Ml  tank,  AH-64A  and  0H-58D 
helicopters)  indicate  that  modern  weapon  systems  require 
comprehensive  tactics  and  operational  technicpaes  development  to 
fully  exploit  weapon  system  improvements.  For  these  tests,  field 
training  was  required  to  fully  develop  operator  skills.  It  is 
envisioned  that  training  demands  on  resources  and  test  facilities 
will  continue  as  next  generation  aircraft,  armored  systems  and 
command  and  control  systems  are  tested.  Multifunction  systems 
will  task  the  limits  of  operators  during  operational  testing. 

b.  In  addition  to  using  models  to  aid  training,  they  are 
inherent  in  the  Training-Modeling  Integration  (T-MI)  methodology 
which  uses  test  data  and  modeling  to  evaluate  crew  and  individual 
performance  during  conduct  of  the  test.  Crew  evaluation 
requirements  are  not  isolated  to  only  FDT&E.  All  types  of 
operational  testing  require  crew  performance  evaluation.  The 
T-MI  methodology  was  developed  within  the  U.S.  Army  Training  and 
Doctrine  Command  (TRADOC)  for  the  investigation  and  development 
of  crew  and  individual  training  information.  M/S  is  used  to 
develop  training  scenarios,  performance  standards,  and  task 
workload  information.  T-MI  has  application  in  addressing  weapon 
system  crew  training  requirements  as  well  as  real  time  crew 
performance  evaluations  during  operational  testing. 

c.  Training  and  crew  performance  analyses  need  detailed  data 
which  identify  when  a  specific  event  occurred  (time  tagged  data) . 
High  resolution  simulations  producing  detailed  time  related 
history  files  provide  an  excellent  source  for  these  type  of 
analyses.  CASTFOREM  (battalion/task  force  level  model)  is  one  of 
the  current  models  used  by  TRADOC  that  produces  comprehensive 
time  related  history  file  information.  More  aggregated  low 
resolution  simulations  are  less  useful  for  training  analysis. 

16-16.  Tactics  and  Techniques  Development 

Use  of  M/S  to  aid  in  the  development  or  refinement  of  tactics  and 
operational  techniques  differs  from  their  use  in  individual 
training  because  the  two  analytical  processes  often  require 
different  simulation  tools.  The  interactive,  man  in  the  loop, 
high  resolution  simulation  is  a  tool  used  for  the  development  of 
tactics  and  operational  techniques.  JANUS,  for  example,  has  been 
used  widely  for  this  application.  The  tester  can  benefit 
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significantly  from  insights  gained  in  the  use  of  M/S  to  examine 
tactics  and  operational  techniques.  Test  design  scenarios  can  be 
reviewed  in  advance  in  order  to  better  understand  weapon  system 
tactics  and  techniques.  It  benefits  the  entire  testing  process 
if  effective  and  well  defined  tactics  are  developed  prior  to 
beginning  the  test. 

16-17.  Test  Planning  and  Operational  Scenario  Development 

a.  Information  available  from  M/S  may  be  used  to  design 
specific  test  events  that  are  in  fact  sensitive  to  critical 
system  operational  issues.  M/S  results  may  aid  decisions 
regarding  sampling,  ordering  of  test  events,  and  scaling. 

Scenario  development  is  a  broad  subject  which  impacts  test  issues 
and  operational  concerns;  terrain  analysis,  tactics,  doctrine, 
pace  of  battle,  etc.  Recent  applications  of  M/S  used  in  scenario 
development  have  concentrated  on  developing  insights  into  test 
exercise  sensitivity  to  terrain,  to  the  number  of  threat  and 
friendly  players,  to  position/location  of  stationary  weapon 
systems,  and  to  the  general  pace  of  battle. 

b.  JANUS  and  CASTFOREM  have  been  used  to  develop  scenario 
information  for  operational  tests  of  the  following  systems; 
Pedestal  Mounted  Stinger  (PMS) ,  Line-of-Site  Forward  (Heavy) ,  and 
M1A2  tank.  In  most  of  the  scenario  development  processes  JANUS 
provided  the  most  utility.  The  ease  with  which  the  modeler  and 
tester  can  incorporate  the  insights  of  proponent  and  crew/ 
operator  into  the  JANUS  simulation  facilitates  scenario 
development. 

16-18.  Data  Extrapolation/Interpolation 

a.  Modeling  may  be  a  cost  effective  supplement  to  operational 
testing  when  test  limitations  and  constraints  are  significant. 

In  order  to  use  M/S  data  for  evaluation  of  a  system  under 
conditions  differing  from  the  actual  test,  it  is  essential  that 
M/S  results  and  test  results  be  compared  to  determine  the 
appropriate  calibration  effort.  Calibration  is  a  careful  and 
intuitive  model/simulation  modification  process  in  order  to 
correlate  model  output  and  test  results.  Calibration  should  be 
undertaken  with  caution.  Changing  model  parameters  to  achieve 
high  correlations  is  only  prudent  if  the  changed  parameters  are 
logical  and  address  intuitively  correct  deviations  from 
established  algorithms.  The  model  accreditation  process  will 
address  calibration.  The  process  will  require  identification  of 
model  parameter  changes  and  the  rationale,  for  such  changes. 

b.  Tester  and  evaluator,  using  an  accredited  model,  have  an 
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opportunity  to  augment  or  supplement  field  test  results  with 
modeled  results.  Extrapolation  can  take  the  form  of  selecting 
different  terrain  or  employing  larger  forces.  Interpolation  can 
take  the  form  of  expanding  sample  sizes  for  trials  that  were 
difficult  or  expensive  to  perform. 

16-19.  Test  Execution 

a.  M/S  applications  in  support  of  test  execution  include  both 
those  of  control  and  actual  conduct.  As  systems  become  more 
complicated  and  include  more  computer  hardware  and  software 
(embedded  or  otherwise) ,  the  challenge  of  executing  a  user  test 
increases.  Automated  stimulation  can  eliminate  large  scripting 
and  controller  cells,  which  simplifies  the  control  task.  Also, 
some  simulations  can  help  the  control  function  by  portraying  the 
overall  status  of  a  scenario  and  isolating  problem  areas,  which 
would  not  be  manageable  by  manual  means,  so  that  test  officers 
can  take  immediate  corrective  actions  when  necessary. 

b.  In  the  conduct  of  a  test,  stimulators,  emulators,  drivers, 
and  simulators  (SEDS)  are  often  the  only  economically  or 
practically  feasible  means  to  conduct  a  large  scale  test. 

Examples  of  SEDS  are: 

Simulator:  Close  Combat  Tactical  Trainer  (CCTT) 

Emulator:  Real  Time  Casualty  Assessment  (RTCA) 

Driver:  Corp  Battle  Simulation 

Stimulator:  Tactical  Simulator 

c.  For  example,  simulators  may  provide  intelligence  sensor 
data  from  echelons  above  corps  (EAC)  during  an  intelligence 
system  test.  Since  personnel  participating  in  test  must  often  be 
trained  on  how  to  operate  complex  equipment  before  that  equipment 
is  available,  this  can  generate  a  need  to  develop  simulator 
training  devices  (integrated  or  standalone)  at  the  same  time  as 
the  system.  Often  such  training  devices  can  serve  multiple 
functions  such  as  simulators,  stimulators,  and  data  collection 
simultaneously.  The  tester  must  coordinate  with  the  combat 
developer  early  in  the  process  and  describe  his  requirements. 

The  same  need  to  capture  data  for  a  test  may  also  be  one  required 
of  the  trainer  (e.g.,  capturing  operator  actions).  As  modeling 
becomes  more  standardized,  extensive  sharing  of  model 
architecture  and  algorithms  could  aid  both  the  combat  developer 
and  the  tester.  Other  uses  of  M/S  are  test  unique  and  critical 
for  testing.  For  example,  M/S  is  often  used  to  provide  Real  Time 
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Casualty  Assessments  (RTCA)  during  the  conduct  of  force-on-force 
testing.  RTCA  may  vary  in  operation  from  simple  look-up  tables 
to  sophisticated  weapon  fly-out  models.  Other  real-time 
applications  for  OT&E  (SEDS)  include  scenario  drivers  which 
generate  events  and  reports  to  information  management  systems  or 
command  elements,  simulators  which  inject  messages  in 
communications  channels,  simulators  of  sensors  or  response 
command  and  control  cells,  flyout  or  endgame  models,  and  loading 
devices  for  software  processors  or  communications  networks.  No 
matter  how  M/S  is  utilized  in  test,  execution  has  the  same 
requirement  to  be  accredited  for  each  application  as  M/S  for  any 
other  use. 

16-20.  Model -Test-Model  (MTM)  Process 

a.  As  introduced  in  paragraph  16-6,  current  acquisition 
policy  promotes  more  effective  linkage  of  analyses  and  testing. 
COEA  and  OT&E  linkage  is  the  objective  of  recent  MTM  initiatives. 
The  MTM  concept  consists  of  a  calibration  scheme  or  use  of 

•observations  to  adjust  model  parameters  or  algorithms.  The  MTM 
process  is  accomplished  through  the  following  steps:  model  the 
scenario;  observe  test  play;  constrain  the  model  to  test 
conditions;  compare  model  measures  to  observations;  adjust  the 
model;  re-model  the  scenario,  and  repeat  as  necessary.  The 
general  process  is  not  new.  References  to  the  basic  components: 
pretest  modeling,  modeling  and  test  comparison,  and  post  test 
modeling,  can  be  documented  in  Army  testing  as  far  back  as  the 
early  1970's. 

b.  The  first  component  of  MTM  is  pretest  modeling.  Pretest 
modeling  estimates  a  range  of  test  results  prior  to  conduct  of 
record  trials/events.  These  results  may  aid  the  tester  in 
supporting  test  design  and  test  scenario  development.  Normally 
the  MTM  pretest  phase  addresses  the  adequacy  of  test 
profiles/scenarios  to  support  the  test  objectives;  e.g.,  are 
expected  ranges  of  engagement  or  ranges  for  communications 
expected  to  be  sampled?  Additionally,  pretest  M/S  can  be  used  to 
make  more  efficient  use  of  test  resources  to  avoid  impractical 
use  of  test  assets.  If  M/S  shows  that  certain  levels  of 
countermeasures  are  expected  to  render  the  test  item  ineffective, 
sufficient  testing  to  define  the  envelope  of  these  levels  must  be 
conducted  to  validate  the  M/S  predictions;  however,  little 
testing  beyond  this  needs  to  be  planned  for  those  conditions.  If 
some  tactically  deployed  targets  are  shown  by  modeling  to  have  no 
reasonable  chance  of  being  in  the  line-of-sight  (LOS)  to  systems 
requiring  LOS,  those  assets  can  either  be  saved  or  used 
elsewhere.  Modeling  or  simulation  can  also  be  used  to  scale 
resources  such  as  targets,  warheads,  or  countermeasures  in  order 
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to  obtain  equivalent  MOE  given  constraints  of  resources,  ranges 
and  test  units.  Modeling  or  simulation  can  be  used  to  train  test 
personnel,  support  test  design  (number  of  trials,  size  of  Blue 
and  Red  forces,  check  execution  timing,  plan  location  of  test 
support  equipment,  validate  threat  surrogates/simulators), 
estimate  key  factors/conditions  which  most  impact  system 
performance,  and  develop  and  refine  test  design  matrices. 

c.  The  second  component  of  MTM  is  comparison  of  modeling  and 
test  results.  This  phase  begins  with  conduct  of  the  operational 
test.  Extensive  work  is  required  to  develop  adequate  operational 
realizations  of  systems  in  combat  models.  The  model  results  and 
test  results  must  be  compared  to  determine  the  significance  of 
differences  that  may  occur.  The  comparison  must  assess  if 
calibration  of  the  model  is  appropriate.  Calibration  should  be 
conducted  when  it  is  determined  that  model  components  must  be 
adjusted  before  any  further  application  of  the  model  will  be 
accredited.  Examples  of  model  components  critical  to 
accreditation  for  T&E  purposes  include:  weapon  system 
algorithms,  man-machine  and  environmental  interfaces,  and  the 
model  scenario  representation. 

d.  The  third  component  of  MTM  is  post  test  modeling.  This 
final  phase  of  the  process  is  the  use  of  the  model  to  make 
additional  estimates.  These  estimates  may  supplement  test 
results.  Issues  for  evaluation  and  the  completeness  of  the  test 
will  determine  exactly  what  modeling  will  be  required.  Listed 
below  are  examples  of  how  models  and  simulations  may  be  used  to 
augment  and  extend  test  results  as  well  as  explain  unexpected 
test  results: 

(1)  Applying  Measures  of  Effectiveness  (MOEs) /Measures  of 
Performance  (MOPs)  to  situations  other  than  those  tested  (running 
many  iterations  based  on  trial  results,  varying  terrain,  varying 
force  sizes) . 

(2)  Investigating  potential  benefits  of  product 
improvement  or  changes  in  doctrine  or  organization. 

(3)  Analyzing  the  sensitivity  of  the  evaluation  findings 
to  known  limitations  in  approximating  realistic  mission  profiles, 
for  example,  types  of  countermeasures  which  could  not  be  played. 

16-21.  Other  Simulation  Sources 

a.  A  supplemental  source  of  simulation  support  for  OT&E  is 
the  Army  Distributed  Interactive  Simulation  (DIS)  efforts.  The 
current  on-line  portion  of  the  DIS  is  SIMulator  NETworks 
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(SIMNET) ,  the  first  simulation  system  to  integrate  large  numbers 
of  manned  vehicle  simulators  on  a  computer  network. 

b.  Battlefield  Distributed  Simulation-Developmental  (BDS-D) , 
the  follow-on  system  for  development  applications,  provides  a 
simulation  capability  that  allows  experimentation  with  human 
system  interaction  in  a  fully  represented  battlefield 
environment. 

c.  The  follow-on  system  for  training  is  the  Combined  Arms 
Tactical  Trainer  (CATT) .  The  first  system  to  be  acquired  under 
the  CATT  program  is  the  Close  Combat  Tactical  Trainer  (CCTT) . 

The  CCTT  will  incorporate  a  man-in-  the-loop  high  resolution 
combat  training  simulator,  useful  in  simulating  multi-system 
interactions  in  a  nonlinear  battlefield  environment. 

d.  Both  BDS-D  and  CATT  have  potential  for  developing  and/or 
testing  new  doctrine  and  tactics  associated  with  developmental 
systems.  They  can  also  be  used  to  measure  the  contribution  to 
system  effectiveness  of  the  human  interaction  with  the 
developmental  system. 
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Table  16-1.  Points  of  Contact  Relative  to  Test  and  Evaluation 

Model  and  Simulation  Support 

a.  Theater  and  above  Level  Force  on  Force  Models  -  focuses 
on  all  force  levels  at  echelons  above  corps.  Includes  multi¬ 
corps,  regional  and  global  models  and  simulations  (e.g..  Force 
Evaluation  Model  (FORCEM) ,  Concepts  Evaluation  Model  V  (CEM  VI) , 
Force  Analysis  Simulation  of  Theater  Administrative  and  Logistic 
Support  (FASTALS) ,  Transportation  Model  (TRANSMO) 

POC:  U.S.  Army  Concepts  Analysis  Agency 

Research  and  Analysis  Support  Directorate 
ATTN:  CSCA-RS 

(301)  295-1692 
DSN:  295-1692. 

b.  Corps  and  Division  Level  Force  on  Force  Models  -  focuses 
on  single  and  multi-division  levels  of  operation  with  or  without 
a  supervising  corps  headquarters  (e.g.,  EAGLE,  Corps  Battle 
Analyzer  (CORBAN) ,  Vector  in  Commander  (VIC) ,  JIFFY,  Command, 
Control,  Communications,  and  Combat  Effectiveness  (FOURCE) . 

POC:  U.S.  Army  TRADOC  Analysis  Command 

Corps/Division  Modeling  and  Analysis  Operations 
Analysis  Directorate  Support  Center 

ATTN:  ATRC-OAC,  Fort  Leavenworth,  KS 
(913)  684-2276 
DSN:  552-2276. 

c.  Brigade  Task  Force  Level  and  Below  Force  on  Force  Models 
-  focuses  on  combined  arms  forces  and  single  functional  elements 
as  they  are  represented  as  integral  parts  of  combined  arms  and 
services  activities  (e.g..  Combined  Arms  and  Support  Task  Force 
Evaluation  (CASTFOREM) ;  JANUS (T),  ELANT,  American  Australian 
British  Urban  Game  (ACABUG) . 

POC:  U.S.  Army  TRADOC  Analysis  Command 

Brigade/Battalion  Modeling  and  Analysis  Support  Center 
ATTN:  ATRC-W,  White  Sands  Missile  Range,  NM 
(505)  678-1012 
DSN:  258-1012. 


d.  Item  System  Level  Performance  Models  -  focuses  on  a 
single  weapons  system  or  piece  of  equipment.  May  address  one  on 
one  or  one  on  many  situations  (e.g..  Air  Defense  Air-to-Ground 
Engagement  (ADAGE) ,  Artillery  Force  Simulation  Model  (AFSM) , 
Simplified  Artillery,  Reliability  Growth  Model,  SESAME, 
Projectile  Effectiveness  Model  (ARTQUIK) ,  TANKWARS,  NATO 
Reference  Mobility  Model) . 

POC:  U.S.  Army  Materiel  Systems  Analysis  Activity 
Special  Studies  and  Activities  Office 
ATTN:  AMXSY-DA 


(410)  278-6576 
DSN:  298-6576. 

e.  Item  System  Level  Weapon  Systems  Performance  data  for 
U.S.  and  threat  systems  and  characteristics  data  for  U.S.  systems 
-  data  focuses  on  the  lowest  level  system  such  as  an  air  defense 
gun  with  its  crew  or  a  tank  with  its  crew  (includes  reliability 
and  supportability) . 

POC:  U.S.  Army  Materiel  Systems  Analysis  Activity 
Special  Studies  and  Activities  Office 
ATTN:  AMXSY-DA 

(410)  278-6576 
DSN:  298-6576. 

f.  Item  System  Level  Weapon  Systems,  operational  and 
characteristic  data  for  threat  systems  -  focuses  on  the 
characteristics  of  lowest  level  threat  system  such  as  an  air 
defense  gun  or  tank. 

POC:  Office,  Deputy  Chief  of  Staff  for  Intelligence 
HQDA  (ODCSINT) 

ATTN:  DAMI-FIT 

(703)  614-8121 
DSN:  224-8121. 

g.  Engineering/Hardware  in  the  Loop  Models  -  focuses  on 
models  and  simulations  which  augment  testing  in  various  stages  of 
the  materiel  acquisition  process.  Models  and  simulations  are 
used  in  investigating  mechanical,  electrical,  and  physical 
phenomena  associated  with  the  functioning  of  an  item  system  in  an 
engineering  sense  (e.g..  Dynamic  Ground  Target  Simulator  (DGTS) , 
Dynamic  Analysis  and  Design  System  (DADS) ,  Wide  Area  Mine  Sublet 
Simulation,  Tow  Weapon  System,  LOSA-T  All  Digital  Six  Degree-of- 
Freedom  (6-DOF)  Simulation  Development  and  Test  Simulation 
(DTSIM) . 

POC(s):  U.S.  Army  Research,  Development  and  Engineering 
(RD&E)  Centers,  U.S.  Army  Communications  -  Electronics 

Command 

ATTN:  AMSEL-RD-AED-MA 

(908)  544-4682 
DSN:  992-4682. 

U.S.  Army  Tank-Automotive  Command 
ATTN:  AMSTA-RYA 

(313)  574-8633 
DSN:  786-8683. 

U.S.  Army  Armament,  Munitions  and  Chemical  Command 
ATTN :  AMSMC-SA 

(201)  724-5262 
DSN:  880-5262. 
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U.S.  Army  Missile  Command 
ATTN;  AMSMI-RD-SS 
(205)  876-4271 
DSN:  746-4271. 

U.S.  Army  Aviation  and  Troop  Command 
ATTN:  AMSAT-B 

(314)  263-1171 
DSN:  693-1171 

POC(s)v  U.S.  Army  Research  Laboratory 
(301)  394-4650 
DSN;  290-4650. 

For  PEO/PM  managed  systems  it  is  common  practice  for  the  PEO  or 
PM  to  sponsor  contractor  M/S  development  related  to  the  T&E  of 
their  individual  systems.  For  systems  of  interest  contact  the 
respective  PEO/PM  to  determine  the  availability  of  M/S  tools  and 
their  state  of  accreditation. 

h.  Vulnerability. 

U.S.  Army  Ballistic  Research  Laboratory 
ATTN;  SLCBR-VL 
(410)  278-1171 
DSN:  298-1171 

i.  Technical  Testing. 

POC:  U.S.  Army  Test  and  Evaluation  Command 
Technical  Director 
ATTN:  AMSTE-TD 

(410)  278-4370 
DSN:  298-4370. 

j.  Operational  Test  and  Evaluation. 

U.S.  Army  Operational  Test  and  Evaluation  Command 
ATTN:  CSTE-MP 

(703)  756-1818 
DSN:  289-1818 

U.S.  Army  Test  and  Experimentation  Command 
ATTN:  CSTE-TDS-IR 

(817)  288-1221 
DSN:  738-1221. 

k.  Model -Test-Model. 

U.S.  Army  TRAEXJC  Analysis  Command  -  WSMR 
ATTN:  ATRC-WMC 

(505)  678-6016 
DSN:  258-6016 


U.S.  Army  Operational  Test  and  Evaluation  Command 
ATTN;  CSTE-MP 
(703)  756-1818 
DSN:  289-1818 

l.  Verification,  Validation,  and  Accreditation  of  Models 
and  Simulations  -  focusses  on  verification  (determined  that  the 
model  and  simulation  accurately  represent  the  developer's 
conceptual  description  and  specifications) ,  validation 
(determination  of  extent  to  which  the  model  and  simulation 
accurately  represent  the  real-world  from  the  perspective  of  the 
intended  use)  and  accreditation  (determination  that  the  model  and 
simulation  is  acceptable  for  its  intended  purpose)  of  those 
models  and  simulations  recognized  as  requiring  management  within 
DA  and  in  accordance  with  AR  5-11,  Army  Model  and  Simulation 
Management  Program  (AMSMP)  (e.g.,  all  models  and  simulations 
identified  by  proponents  as  requiring  V,V&A  under  management  of 
the  AMSMP) . 

POC;  HQ  DA 

U.S.  Army  Model  Improvement  and  Study  Management 
Agency 

ATTN:  SFUS-MIS 

(703)  746-8072 
DSN:  286-8072. 

m.  Man-in-the-loop  Training  Simulators. 

POC;  U.S.  Army  Simulation,  Training,  and  Instrumentation 
Command 

ATTN:  AMSTI-TD 

(407)  380-4325 
DSN;  960-4325. 


Table  16-2.  Credibility  Issues  to  Address 
in  Using  Models  and  simulations  to  Support  T&E 

Credibility,  as  applied  to  modeling  and  simulation  processes 
and  results,  is  a  combined  impression  of  the  inputs,  processes, 
outputs,  conclusions,  persons  or  agencies  involved,  and  the 
strength  of  the  evidence  presented.  This  appendix  contains 
questions  that  the  developing,  review,  and  user  communities 
should  ask  and/or  be  prepared  to  answer  when  models  and 
simulations  are  used  to  support  Test  and  Evaluation. 

1.  Issues  relating  to  Verification,  Validation,  and 
Accreditation  (WA)  of  Models  and  Simulations  (M/S) . 

a.  Has  the  M/S  gone  through  an  approved  process  to 
establish  its  credibility  for  this  specific  application? 

b.  Were  M/S  results  compared  with  combat,  field  test,  and 
other  models?  If  so,  what  were  the  results? 

c.  What  have  the  results  been  validated  against? 

d.  What  is  the  availability,  date,  and  source  of  data?  Is 
data  available  to  support  system  requirements?  If  not,  what 
assumptions  will  be  made?  How  will  critical  variables  be 
represented?  Is  there  imbedded  data  in  the  model?  If  so,  are 
those  variables  documented  and  is  the  data  defined?  Who  has 
reviewed  and  certified  the  use  of  this  data  for  the  study? 

e.  How  robust  are  the  results  on  operational  capability  and 
supportability? 

f.  Who  built  the  model? 

g.  Who  certified  model  inputs? 

h.  Who  certified  the  tactics/scenarios  or  changes  to 
existing 

scenarios? 

i.  Who  did  the  verification  and  validation? 

j.  What  implicit  and  explicit  assumptions  were  made?  What 

are 

the  model  limitations? 

k.  Is  there  sufficient  documentation  of  the  model,  its 
assumptions,  data  requirements  and  methodology? 

l.  What  sensitivity  analyses  have  been  performed? 

m.  How  far  has  the  model  been  pushed  to  extremes  and  how 
has  it  performed?  Has  the  M/S  domain  been  established? 


n.  What  field  test  results  have  been  fed  back  into  the 
model  for  validation? 

o.  Is  there  a  documented  audit  trail  of  data,  methodology 
and  code  changes,  and  scenario  changes?  Will  it  provide 
traceability  of  critical  decisions? 

p.  Who  maintains  the  model? 

q.  What  is  the  source  of  threat  data?  Is  it  consistent 
with  data  used  in  other  analyses?  What  is  the  source  of  threat 
(RED)  tactics  used  in  the  scenario? 

r.  What  variables  of  the  operational  environment  are  not 
represented? 

2.  Issues  relating  to  the  study  concept. 

a.  Why  was  this  model  used  in  lieu  of  testing?  What 
specific  questions  are  to  be  addressed  by  the  model? 

b.  Was  M/S  discussed  in  the  TEMP? 

c.  Did  the  simulation  accurately  reflect  the  system 
requirement  and  any  available  developmental  test  data? 

d.  What  is  the  linkage  between  TT&E  and  OT&E  and  modeling? 

e.  Why  was  this  particular  model  chosen?  What  was  it 
designed  to  do?  What  are  its  strengths/weaknesses?  Where  has  it 
been  used  before? 

f.  Is  there  adequate  funding  to  support  the  M/S?  By  whom? 
Is  the  M/S  cost-effective? 

g.  What  elements  of  M/S  should  be  confirmed,  by  operational 
testing? 

h.  Were  excursions  conducted  on  critical  variables  and 
system  parameters?  If  so,  why  and  what  were  they? 

i.  What  impact  (if  any)  did  the  excursions  have  on  the 
evaluation? 


j .  What  is  the  degree  of  independence  of  modelers  with 
respect  to  the  program  office,  material  developer  and  hardware 
contractor?  If  the  M/S  developer  is  associated  with  the  program 
office,  who  conducted  the  independent  assessment  of  the  M/S 
applicability? 

k.  Has  this  model  been  used  by  the  developer  of  the  weapon 
system?  What  were  the  results? 


l.  Who  is  expected  to  use  or  operate  the  model? 

m.  Can  the  model  be  designed,  built  and/or  modified  faster 
and/or  cheaper  than  the  system  it  represents? 

n.  If  multiple  models  are  used,  what  are  the  linkages? 

What  are  the  data  structures? 


Table  16-3.  Examples  of  the  Application  of  Modeling  and 
Simulation  in  Test  &  Evaluation 

1.  To  support  pretest  planning. 

2.  To  assist  in  the  identification  of  critical  issues  to  be 
addressed  in  a  test. 

3.  To  identify  important  test  parameters  earlier. 

4.  To  grossly  bound  the  problem  and  proposed  solutions  based  on 
the  intended  environment,  force  structure,  threat,  tactics, 
strategy,  and  doctrine. 

5.  To  identify  oversights  and  flawed  logic. 

6.  To  determine  sensitivity  of  a  program  to  various  input 
parameters . 

7.  To  conduct  non— destructive  evaluations  of  high  cost  items 
which  would,  by  their  nature,  be  destroyed  in  actual  hardware 
tests. 

8.  To  provide  better  understanding  when  full-scale  testing  is 
not  possible.  To  augment,  extend,  and  enhance  test  results,  in 
general . 

9.  To  provide  multiple  "environments”  for  examining  test 
questions. 

10.  To  provide  advantages  of  test  compression,  control 
expenditures,  provide  for  replicability,  and  reduction  of 
variables  under  study. 

11.  To  assess  impact  of  known  parameters  of  unavailable  threat 
systems . 

12.  To  accomplish  human  factors  supportability  evaluations  in 
part-task  or  limited  fidelity  "mock-ups". 

13.  To  provide  estimates  of  potential  test  outcomes. 

14.  To  extrapolate,  with  caution,  test  results  into  other 
scenarios  and  levels  of  force  aggregation. 

15.  To  address  issues  which  cannot  be  physically  tested. 

16.  To  address  "what  if”  questions  during  post-test  analyses. 

17.  To  develop  and  refine  test  scenarios  and  data  matrices  to 
obtain  maximum  data  from  limited  test  resources. 

18.  To  develop  new  tactics  for  the  employment  of  new  weapon 
systems  under  test. 
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19.  To  provide  overall  system,  scenario,  or  environment 
representation. 

20.  To  represent  the  input,  process,  and  output  of  non-available 
systems,  subsystems,  or  components  (friendly,  or  threat) . 

21.  To  represent  the  whole  integrated  system  when  all  components 
are  not  available. 

22.  To  allow  an  assessment  of  test  events  that  would  otherwise 
be  exposed  to  threat  intelligence  exploitation. 

23.  To  act  as  a  system  driver  or  stimulator  in  order  to  stress  a 
system  beyond  available  field  test  scenarios. 

24.  To  determine  adequacy,  effectiveness,  and  suitability  of  the 
planned  operational,  maintenance,  and  supportability  concepts. 

25.  To  estimate  mature  system  mission  reliability,  availability, 
and  logistics  support  frequency. 
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Chapter  17 

Test  Incidents  and  Related  Reporting 


17-1.  General 

a.  This  chapter  will  discuss  policies,  procedures,  and 
responsibilities  for  the  reporting  of  test  results,  corrective 
actions,  and  closures  to  enable  the  continuous  evaluation  process 
to  function.  The  ultimate  purpose  is  to  ensure  that  test  results 
are  made  available  on  an  expedited  basis  for  evaluators,  program 
executive  officers,  and  decision  makers  to  analyze  emerging  data 
and  assess  system  performance.  The  term  "tester”  used  in  this 
chapter  refers  to  the  on-site  organization  actually  conducting  or 
monitoring  the  test  and  collecting  and  reporting  the  data. 

b.  The  Test  Incident  Report  (TIR)  is  the  means  by  which  the 
minimum  essential  data  for  test  incidents,  their  respective 
corrective  actions,  closures,  and  other  test  information  are 
reported.  The  TIR  form  (DA  Form  XXXX-R)  is  shown  in  Figure  17-1. 
The  TIR  contains  two  types  of  data.  One  type  consists  of  test 
incident  (TI)  data.  The  tester  prepares  this  data.  The  tester 
omits  section  VI  when  preparing  the  TIR.  The  other  consists  of 
corrective  action  (CA)  and  closure  data  which  are  prepared  by  the 
program  manager.  These  two  data  types  are  merged  together  by  the 
Army  Test  Incident  Reporting  System  (ATIRS)  at  Aberdeen  Proving 
Ground,  MD,  which  is  administered  by  the  U.S.  Army  Combat  Systems 
Test  Activity  of  the  U.S.  Army  Test  and  Evaluation  Command. 

(INSERT  FIGURE  17-1) 

17-2 .  Requirements 

a.  Materiel  developers,  combat  developers,  evaluators,  and 
other  organizations  participating  in  the  acquisition  process  must 
be  informed  of  system  performance  during  tests  in  a  timely  manner 
so  that  test  information  is  available  for  immediate  use  by 
acquisition  community  members,  corrective  actions  initiated,  and 
the  continuous  evaluation  of  the  systems  enhanced. 

b.  The  TIR  is  used  as  the  medium  to  provide  the  results  of 
any  incident  occurring  during  test  and  to  report  corrective 
actions  taken.  Also,  the  TIR  reports  the  results  of  subtests  and 
serves  as  an  interim  report,  when  a  formal  test  report  has  not 
been  published.  TIRs  will  be  completed  using  the  instructions 
provided  in  Figures  17-2  and  17-3. 

(INSERT  FIGURE  17-2) 
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(INSERT  FIGURE  17-3) 

c.  Incident  reports,  maintenance  data,  and  corrective  action 
information  are  also  required  by  Reliability,  Availability,  and 
Maintainability  (RAM)  scoring  and  assessment  conference  members 
to  form  the  basis  for  the  assessment  of  RAM  per  AR  702-3  and 
integrated  logistics  support  (ILS)  per  AR  700-127. 

d.  The  test  incident  and  corrective  action  data  in  the  TIR 
will  be  added  into  ATIRS.  ATIRS  provides  an  Army  standard  method 
of  electronically  exchanging,  storing,  processing,  and  reporting 
data  on  results  of  testing,  their  corrective  actions,  and  other 
test-related  information.  ATIRS  is  used  for  the  storage  of  all 
test  incidents,  corrective  actions,  and  closure  information. 
Assistance  on  ATIRS  is  available  by  electronic  mail  through  the 
Defense  Data  Network  (DDN)  at  atirs@csta-emhl.apg.army.mil  or  by 
submitting  a  request  to  Commander,  U.S.  Army  Combat  Systems  Test 
Activity,  ATTN:  STECS-DA-CA,  Aberdeen  Proving  Ground,  MD 
21005-5059. 

e.  The  tester  prepares  TIRs  for  all  government,  required 
contractor  tests  and  for  Production  Phase  tests  which  support  a 
materiel  release  decision.  TIRs  may  also  be  prepared  for  other 
tests  as  required  by  the  program  manager  or  other  test  sponsors. 

f.  The  program  manager  prepares  corrective  action  and  closure 
data  reports  for  input  into  ATIRs  for  critical,  major,  and 
major-linked  repetitive  minor  TIRs  as  a  minimum.  All  TIRs  must 
be  considered  for  corrective  action,  and  the  TIR  should  reflect 
action  taken  with  supporting  rationale.  After  consideration  the 
decision  may  be  that  no  corrective  action  is  required  for  minor 
TIRs,  the  intent  is  to  produce  a  better  system. 

g.  Production  and  post  production  tests  of  ammunition  are 
excluded  from  submission  of  TIRS.  The  reporting  procedures  in  AR 
75-1  are  used  for  these  items. 

17-3.  Security 

a.  Since  the  TIR  data  will  be  transmitted,  stored,  and 
interactively  accessed  via  unsecured  media,  care  must  be  taken  to 
ensure  that  documents  provided  to  ATIRS  contain  no  classified 
information. 

b.  In  the  event  that  information  pertaining  to  a  test 
incident  is  classified,  the  information  will  be  published 
separately  in  a  classified  TIR  and  distributed  per  the  listing 
agreed  to  by  TIWG  members.  Additionally,  an  unclassified  report 
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will  be  provided  to  ATIRS,  in  the  data  format  of  Figures  17-4, 
17-5,  and  17-6,  as  applicable,  which  references  the  TIR  that 
contains  the  classified  information.  AR  380-5  provides 
instructions  on  handling  classified  documents  from  automated 
equipment.  Since  portion  markings  are  not  possible  on  the  TIR, 
the  individual  blocks  in  a  classified  TIR  need  not  be  marked 
provided  that; 

(1)  Classification  markings  are  placed  top  and  bottom. 

(2)  A  statement  is  included  in  Block  90  showing  the  source 
of  the  classification,  full  address  of  proponent  and 
declassification  date/event/Originating  Agency's  Determination 
Required  (OADR) . 

(3)  A  statement  is  provided  in  Block  90  listing  the 
classified  block  numbers  and  their  classification  levels.  In 
addition,  a  statement  will  be  provided  to  indicate  that  other 
blocks  not  listed  are  unclassified. 

(INSERT  FIGURE  17-4) 

(INSERT  FIGURE  17-5) 

(INSERT  FIGURE  17-6) 

c.  It  is  the  responsibility  of  the  originators  and  recipients 
to  safeguard  the  classified  information  per  AR  380-5. 

d.  The  program  manager  should  address  Operations  Security 
(OPSEC)  and  Competition  Sensitive  (CS)  implications  of  TIR 
information  prior  to  commencement  of  pretest  activities.  If  the 
reports  are  expected  to  contain  OPSEC  information,  the  program 
manager  will  notify  the  document  originator,  and  the  ATIRS 
administrator  of  any  limits  to  be  placed  on  content,  electronic 
mail  distribution,  storage,  or  interactive  access  per  AR  530-1. 
Similar  procedures  will  be  followed  for  reports  expected  to 
contain  proprietary  or  CS  information. 

e.  Access  to  the  ATIRS  database  is  requested  through  the 
ATIRS  administrator.  As  a  default,  government  users  will  have 
open  access  to  the  ATIRS  database,  unless  the  data  is  restricted 
by  the  program  manager  or  tester.  All  contractors  are  restricted 
to  those  data  authorized  by  the  program  manager  or  tester.  The 
T&E  Manager  will  have  access  to  all  data  associated  with  their 
commodity  command. 

17-4.  Test  Integration  Working  Group  (TIWG)  Actions 
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a.  The  TIWG  plays  an  active  role  in  developing  the  T&E 
program  and  integrating  various  disciplines  and  interest. 
Therefore,  the  TIWG  is  used  as  the  medium  to  effect  necessary 
actions  crucial  to  the  TIR  process. 

b.  To  ensure  consistency  of  terms  across  test  phases  and 
milestones,  prior  to  any  TIWG  the  program  manager  and  tester  will 
contact  ATIRS  either  by  mail,  electronic  mail  (see  paragraph 
17-2d  for  mail  and  electronic  mail  addresses) ,  or  dial-in/TELNET 
(after  receiving  access  authorization)  for  a  list  of  possible 
values  for  the  TIR  blocks  shown  in  paragraphs  17-4c  and  17-4d. 

The  list  will  form  the  basis  for  agreement/understanding  of 
values  at  TIWGs  as  discussed  below. 

c.  At  the  initial  TIWG  meeting,  the  PM,  MATDEV  and  tester 
will  discuss  initiate  the  following. 

(1)  Identify  all  tests  which  provide  data  to  support  a 
milestone  decision.  These  tests  are  then  reflected  in  the  TEMP 
and  TIRs  will  be  prepared  for  those  tests. 

(2)  Establish  acceptable  unique  values  for  the  below 
blocks  so  that  the  ATIRS  database  can  be  set  up  for  the  program 
(block  details  in  Figure  17-2) . 

(a)  Test  Title  (Block  2) . 

(b)  System  (Block  7) . 

d.  Prior  to  test,  as  the  program  develops,  the  program 
manager  and  tester  will  lead  the  following  actions  in  subsequent 
TIWGs: 


(1)  Establish  unique  values  to  be  registered  with  ATIRS 
for  the  following  blocks: 


(a) 

Test  Agency 

(Block  5) . 

(b) 

Test  Sponsor 

(Block  6) . 

(c) 

Model  (Block 

10)  . 

(d) 

Manufacturer 

(Block  13) . 

(e) 

Contract  NO. 

(Block  14) . 

(f) 

Subsystem  (Block  31) . 
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(g)  Failure  Definition/Scoring  Criteria  Classification 
(Block  42) . 

(h)  Chargeability  (Block  43) . 

(2)  Establish  the  format  and  units  of  measure  to  be 
registered  with  ATIRS  for  the  following  blocks: 

(a)  Test  Life:  Units  (Blocks  21-25) . 

(b)  Part  Life:  Units  (Blocks  62-64) 

(3)  Discuss  possible  data  values  for  use  during  test  as 
based  on  the  ATIRS  listing  for  the  below  blocks.  Register  any 
additional  values  with  ATIRS.  The  tester  may  add  other  values 
during  conduct  of  test  if  the  ATIRS  listing  is  insufficient  and 
provided  the  values  are  registered  with  ATIRS. 

(a)  Action  (Blocks  34  and  57) . 

(b)  Categories  (Block  46) . 

(c)  Keywords  (Block  47). 

(d)  Test  Environment;  Type;  Condition  (Block  48). 

(e)  Disposition  (Block  49) 

(f)  Type/Level  Used/Level  Prescribed/Level  Recommended 
(Blocks  80-83) . 

(4)  Discuss  security  guidance  and  procedures  on  data 
handling. 

(5)  Determine  authorizations  and  data  restrictions  (if 
competition  sensitive  data  are  involved)  to  ATIRS.  Fill  and 
submit  the  ATIRS  access  authorization  form  shown  in  Figure  17-7. 

(INSERT  FIGURE  17-7) 

(6)  Establish  a  distribution  list  for  the  TI  and  CA  data 
for  users  preferring  data  to  be  sent  directly  from  the  tester . 

The  list  will  include  format  (data  stream,  TI  form  format,  etc.), 
distribution  method  (computer  transfers,  electronic  mail,  floppy 
disk,  hardcopy,  etc.),  and  addresses,  including  electronic 
mailbox  addresses.  Users  to  consider  include  the  program 
manager,  both  developmental  and  operational  independent 
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evaluators/assessors,  logistician,  combat  developer,  and  T&E 
Database  Manager. 

(7)  Determine  recipients  of  hard  copy  information,  such  as 
classified  photographs  or  other  information  related  to  a  TIR. 

(8)  Determine  capabilities/procedures  of  participants  in 
implementing  provisions  of  this  pamphlet  (e.g.,  how  contractor 
TIRs  are  processed  for  input  to  the  independent 
evaluators/assessors  and  ATIRS  administrator) . 

(9)  Determine  data  collection  procedures  for  all  of  the 
test  and  commodity-unique  additions. 

17-5.  Notifying  Database  Personnel 

a.  After  the  TIWG,  the  program  manager  in  coordination  with 
the  tester  must  then  register  the  TIWG-agreed  acceptable  values 
for  the  specified  blocks  in  paragraph  17-4  with  the  ATIRS 
administrator  prior  to  commencement  of  testing.  Registration  is 
accomplished  through  either  electronic  mail,  facsimile,  or  in 
writing  to  the  ATIRS  administrator. 

b.  All  additions  to  the  blocks  in  the  TIR  or  changes  to  the 
TIWG-agreed  values  must  be  coordinated  with  the  ATIRS 
administrator  so  that  consistent,  readily  identifiable  data  is 
stored,  retrieved,  and  used. 

17-6.  Test  Incident  Data 

a.  The  tester  (government  or  contractor)  prepares  the  TIR 
test 

incident  data  (i.e.  Header  blocks  1-8  and  Sections  I  to  V  of  DA 
Form  XXXX-R) .  ("TIR"  used  in  this  section  17-6  refers  to  just 
the  test  incident  data  sections.)  Instructions  for  completion  of 
TIR  input  are  reflected  below.  Additionally,  Figure  17-2 
provides  detailed  instructions  pertaining  to  each  individual  TIR 
block. 


b.  A  TIR  is  prepared  for  each  test  incident  occurring  on  an 
identifiable  test  item/system,  without  regard  to  the  number  of 
times  the  test  incident  recurs.  Some  groupings  of  incidents  are 
authorized  for  minor  or  extremely  frequent  occurrences  that  do 
not  impact  mission  reliability.  When  an  incident  involves  a 
problem  (such  as  an  inherent  operational  defect/safety/HFE 
problems) ,  which  does  not  require  maintenance  and  can  be 
determined  by  inspection  or  examination  to  be  common  to  all 
samples  of  the  test  item  that  are  accessible  to  the  tester,  the 
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tester  may  prepare  a  single  TIR  that  addresses  the  problem,  in 
lieu  of  a  TIR  for  each  test  item. 

c.  A  TIR  is  prepared  for  test  incidents  involving  government- 
owned  products,  such  as  items  covered  by  a  warranty  or 
government-furnished  equipment.  The  MATDEV  item  manager  will 
prepare  a  Quality  Deficiency  Report  (QDR) ,  based  on  the  TIR 
input,  per  AR  702-7-1.  A  separate  QDR  will  not  be  prepared  by 
the  tester. 

d.  A  TIR  will  be  prepared  whenever  the  need  arises  during 
pretest,  test,  or  post-test  activities  to  report — 

(1)  Non-receipt  of  all  or  part  of  any  applicable  test 
support  package  or  an  inadequacy  in  the  components  of  a  support 
package,  in  particular,  the  System  Support  Package.  Also,  a  TIR 
will  be  prepared  if  the  System  Support  Package  Component  List  is 
incomplete. 

(2)  Start  of  test,  to  establish  a  record  of  the  test  start 
date  and  major  component  serial  numbers  (e.g.,  engine, 
transmission,  etc.)  and  the  starting  hours  for  the  major 
components . 

(3)  Receipt  of  materiel  in  unsatisfactory  condition  for 

test. 

(4)  Any  functional  area  characteristic,  defect,  or 
discrepancy,  actual  or  incipient,  that  affects,  may  ultimately 
affect,  or  pertains  to  health,  safety,  environmental,  operational 
suitability  or  effectiveness,,  or  compliance  with  contract 
specifications  or  requirements  documents  of  the  item/system  to 
include  its  hardware,  operator/crew  and  maintenance  personnel, 
prescribed  training,  publications,  tools,  diagnostic  and  support 
equipment,  and  associated  software. 

(5)  The  need  for  or  accomplishment  of  a  scheduled 
preventive  maintenance  check  and  service  if  the  maintenance  data 
associated  with  the  task  is  to  be  scored  as  chargeable  and 
scheduled  and  is  to  be  used  in  the  computation  of  maintainability 
statistics  for  the  test. 

(6)  The  need  for  or  accomplishment  and/or  application  or 
installation  of  a  modification  to  an  end  item  or  it's  associated 
software.  Indication  will  be  made  in  block  90  of  the  effects  on 
previously  reported  test  conditions. 

(7)  The  need  for  installation,  removal,  adjustment. 
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repair,  or  replacement  of  a  component,  assembly,  or  software  for 
reasons  other  than  above. 

(8)  The  accomplishment  of  off-item  component  or  assembly 
repair,  whether  accomplished  by  the  tester  or  by  the 
contractor/manufacturer,  on  or  off  the  test  site,  if  such 
maintenance  is  not  reported  with  the  basic  incident. 

(8)  End  of  test,  to  establish  a  record  of  the  test  end 
date  and  the  ending  hours  for  the  major  components. 

e.  In  addition,  TIRs  are  used  to  report  the  following; 

(1)  A  summarization  of  subtest  results  (performance, 
safety/HFE,  etc.).  In  Block  90,  a  caution  "Preliminary  Data  — 
Subject  to  Further  Review"  leads  into  the  following  format  of 

information:  "a.  Reference  Test  Plan,  subtest  _ ,  paragraph, 

_ t  dated  _ .  b.  Summary  of  Results,  c.  Abbreviated 

Analysis."  The  program  manager  or  evaluators  may  request 
additional  data  to  be  in  the  TIR  if  needed. 

(2)  The  achievement  of  important  milestones  in  the  test 
program,  such  as  receipt  or  shipment  of  the  test  item(s)  or 
commencement  or  completion  of  testing,  or  a  specific  phase  of 
testing. 

f.  Each  TIR  will  be  assigned  a  TIR  classification  value  by 
the  tester  that  reflects  the  degree  of  seriousness  of  the 
reported  incident  or  test  findings,  regardless  of  cause, 
frequency,  or  expected  probability  of  occurrence.  The  four 
acceptable  TIR  classification  values  are:  Critical,  Major, 
Minor,  and  Information  and  are  described  as  follows: 

(1)  Critical — 

(a)  Involves  a  catastrophic  or  critical  hazard  related 
to  health  or  safety  of  personnel  (death  or  severe  injury  or 
occupational  illness;  Categories  I  and  II  per  MIL-STD-882) . 

(b)  Involves  a  catastrophic  safety  hazard  to  the 
item/ system  under  test  (unplanned  system  loss;  Category  I  per 
MIL-  STD-882). 

(c)  Reports  test  results  which  make  test  suspension  or 
termination  advisable. 

(2)  Major — 
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(a)  Involves  a  marginal  hazard  to  health  or  safety  of 
personnel  (Category  III  per  MIL-STD-882) . 

(b)  Involves  a  critical  safety  hazard  to  the 
item/system  under  test  (unplanned  major  system  damage;  Category 
II  per  MIL-STD-  882). 

(c)  Reports  the  inability  of  the  test  materiel 
(including 

diagnostic  equipment,  tools,  publications,  software,  etc.)  to 
meet  a  critical  or  essential  functional  area,  design,  or 
performance  requirement. 

(d)  Reports  subtest  results  which  reflect  inadequate 

per¬ 

formance. 

(e)  Involves  two  or  more  repetitive  minor  TIRs  in  which 
their  cumulated  effect  could  result  in  any  of  the  above  four 
conditions. 

(3)  Minor — 

(a)  Reflects  an  actual  or  incipient  malfunction, 

defect, 

hazard,  or  negative  finding  that  does  not  qualify  as  critical  or 
major. 


(b)  Reports  subtest  results  which  reflect  marginal 
performance. 

(4)  Information — reports  modification  to  the  tested  item, 
current  condition  of  the  tested  item,  test  findings,  subtest 
results,  safety  release  information,  or  other  types  of 
information. 

g.  If  the  cumulated  effect  of  two  or  more  repetitive  minor  TIR 
incidents  exhibiting  the  same  manifestations  meets  the  definition 
for  a  major  TIR,  then  a  major  TIR  can  be  written  listing  the 
related  TIRs.  However,  each  incident  will  still  be  reported 
separately.  As  additional  repetitive  incidents  occur,  this  major 
TIR  will  be  revised  to  cover  the  additional  incidents.  Since  it 
is  a  derived  major  TIR,  indicate  "DERIVED  MAJOR"  in  Block  47. 

h.  A  change  or  addition  to  information  contained  in  a 
distributed  TIR  (such  as  a  more  complete  analysis,  a  description 
of  deferred  maintenance,  TIR  reclassification,  incorporation  of 
scoring  conference  results,  or  addition  of  any  other  data  that  is 
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required  to  complete  or  update  the  TIR)  will  be  accomplished  by 
issuing  revision (s)  to  the  original  TIR.  The  revision  will 
replace  the  original  TIR  (or  previous  revision(s))  in  ATIRS  and 
in  any  other  files  (manual  or  otherwise)  that  may  be  created  in 
ATIRS . 

i.  In  revising  previously  submitted  TIR  data,  the  original 
data  must  be  accounted  by  reporting  in  block  90  of  the  TIR  the 
information  which  has  been  revised.  Figure  17-2  contains 
instructions  on  how  to  fill  in  the  revision  information  in  Block 
90. 


j.  The  basic  TIR  number  is  not  to  be  altered;  however,  block 
1  provides  for  identifying  the  revision  number  and  date. 

k.  In  those  instances  where  the  TIR  test  incident  data  is 
revised  to  change  the  TIR  incident  classification  (block  32), 
block  90  must  provide  rationale  for  the  change. 

l.  The  tester  will  electronically  transmit  the  TI  data  and 
revisions,  if  possible,  by  dial-in  or  TELNET  (provided  ATIRS 
access  is  authorized)  or  by  electronic  mail 

(atirs@csta-emhl.apg.army.mil)  to  ATIRS  using  the  data  stream 
specified  in  Figure  17-5.  If  a  data  stream  is  not  possible,  then 
the  TIR  form  of  Figure  17-4  (excluding  Section  VI)  may  be 
transmitted.  No  hardcopy  TIRs  are  to  be  submitted  to  ATIRS.  In 
addition,  the  AMC  commodity  command  T&E  Manager  will  be 
distributed  the  TI  data  lAW  Figure  17-4  by  electronic  mail.  Data 
will  also  be  distributed  to  other  users  per  agreements  reached  by 
TIWG  members . 

m.  In  those  instances  where  electronic  transmission 
capability  does  not  exist,  tape,  floppy  disk  or  other  electronic 
storage  media  of  the  test  incident  or  corrective  action 
information  will  be  forwarded  by  the  preparer  to  ATIRS  (same 
address  listed  in  paragraph  17-2d)  for  inclusion  in  the  data 
base.  Media  compatibility  must  be  verified  with  the  ATIRS 
administrator.  Hardcopy  TIRs  are  not  to  be  submitted  to  ATIRS. 

n.  Distribution  of  TIRs  that  are  prepared  for  tests  other 
than  those  identified  by  the  TIWG  is  limited  to  the  addressees 
designated  by  the  PM,  MATDEV  or  the  tester. 

o.  The  program  manager  will  prepare  a  listing,  using 
agreements  reached  by  the  TIWG  members,  for  distribution  of 
photographs  and  classified  TIRs. 

p.  Until  an  automated  support  system  is  established  to 
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efficiently  process  pictures  and  graphics,  transmission  of 
pictures  and  graphics  by  facsimile  is  encouraged. 

q.  Timeliness  of  TIRS.  To  ensure  timely  receipt  of  TIRS,  the 
tester  will  prepare  and  initiate  distribution  as  follows: 

(1)  Critical  test  incidents.  The  tester  notifies  the 
program  manager  by  telephone  within  24  hours  after  detection  of 
the  incident.  The  tester  prepares  the  TIR  for  distribution  as 
soon  as  the  data  has  been  validated.  Critical  TIRs  are 
transmitted  electronically  to  the  program  manager,  T&E  Manager, 
higher  headquarters  test  manager,  logistician,  both  the 
developmental  and  operational  independent  evaluators/assessors, 
and  the  ATIRS  administrator.  Electronic  message  notification 
does  not  negate  the  requirement  for  accident  reporting  per  AR 
385-40. 

(2)  Major,  Minor,  and  Information  test  incidents.  The 
tester  prepares  and  distributes  the  TIR  as  soon  as  the  data  has 
been  validated,  after  detection  of  the  incident  or  completion  of 
the  subtest. 

(3)  Revisions  to  TIRs  are  distributed  as  soon  as  possible 
after  validation  of  the  data  by  the  Data  Authentication  Group 
(DAG)  . 


r.  When  the  test  materiel  is  received,  if  the  tester  feels  it 
is  in  an  unsatisfactory  condition  for  testing,  and  believes  the 
condition  may  jeopardize  test  objectives,  invalidate  test 
results,  or  render  testing  unsafe,  the  tester,  after  coordination 
with  higher  headquarters  test  manager,  should  notify  the  materiel 
developer  by  telephone. 

(1)  If  corrections  can  readily  be  made  with  no  delay  in 
scheduled  test  initiation,  the  tester,  after  coordination  with 
the  higher  headquarters  test  manager,  should  obtain  telephonic 
concurrence  from  the  program  manager  and  initiate  corrective 
actions  or  repairs.  This  means  being  able  to  place  the  item  or 
system  in  serviceable  condition  in  accordance  with  the  contract 
specification  and  standards  using  available  maintenance  or  repair 
capabilities.  A  major  TIR  will  be  written. 

(2)  If  corrections  cannot  readily  be  made,  the  tester, 
after  coordination  with  the  higher  headquarters  test  manager, 
should  recommend  by  phone  rescheduling,  suspension,  or 
termination  of  the  test.  If  the  test  is  delayed  he  should 
request  disposition  instructions  for  the  test  items  from  the 
materiel  developer  and  prepare  a  critical  TIR. 
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17-7  Corrective  Action  Data 

a.  The  program  manager  prepares  the  corrective  action  data 
(Section  VI  of  DA  Form  XXXX-R)  for  all  TIRs  per  the  instructions 
contained  below.  Additionally,  Figure  17-3  provides  detailed 
instructions  pertaining  to  each  individual  corrective  action  data 
block. 


b.  The  information  will  reflect  the  program  managers  analysis 
of  the  problem,  status,  and  a  description  of  corrective  action  or 
a  report  that  no  corrective  action  is  proposed,  as  long  as 
adequate  justification  is  provided. the  information.  Corrective 
action  data  will  be  prepared  with  the  best  information  available 
at  the  time  of  preparation,  even  though  the  information  may  be 
incomplete. 

c.  Whenever  possible,  the  program  manager  should  implement 
the  necessary  corrective  actions  during  the  conduct  of  the 
planned  test  program.  This  provides  the  evaluator/assessor  the 
opportunity  to  analyze  the  corrective  action  and  determine  the 
need  for  any  additional  testing,  minimizing  the  need  for 
unplanned  additional  verification  tests  or  commencement  of  a  new 
acquisition  phase  with  corrective  actions  of  unknown  adequacy. 
During  OT,  the  configuration  is  fixed  and  corrective  actions 
normally  are  not  implemented  during  its  conduct.  If  a  corrective 
action  is  implemented  during  testing,  a  TIR  will  be  written. 

d.  Each  corrective  action  taken  is  assigned  a  classification 
value  that  reflects  the  status  of  the  corrective  action.  The 
classification  is  determined  by  a  corrective  action  review  team 
lAW  the  procedures  of  paragraph  17-8  below.  The  four  acceptable 
corrective  action  status  classifications  are  Open,  Completed, 
Verified,  and  Not  Required  and  are  defined  as  follows: 

(1)  Open — corrective  action  is  required  but  has  not  been 
verified  or  no  corrective  action  has  been  developed. 

(2)  Verified — corrective  action  has  not  been  completed. 
Corrective  action  is  required  and  a  proposal  has  been  identified, 
but  the  corrective  action  has  not  been  completed. 

(3)  Completed — corrective  action  has  been  completed. 
Corrective  action  is  required  and  has  been  proposed,  considered 
appropriate,  implemented,  verified  by  test  or  analysis  as  being 
satisfactory,  evaluated  for  effectiveness,  and  approved  for 
production. 

(4)  Not  Required — corrective  action  is  not  required. 
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e.  The  initial  corrective  action  data  will  be  submitted  to 
the  ATIRS  administrator  within  60  days  of  the  date  reflected  in 
the  TIR  release  date  (block  1  of  the  TIR) .  Subsequent  updates 
are  submitted  as  appropriate. 

f.  A  change  or  addition  to  previously  distributed  corrective 
action  infojrmation  to  ATIRS  is  made  through  submission  of  revised 
data.  The  revised  data  replaces  the  original  corrective  action 
information  in  ATIRS. 

g.  The  corrective  action  data  will  be  electronically 
transmitted  by  dial-in  or  TELNET  (provided  ATIRS  access  is 
authorized)  or  by  electronic  mail  (atirs@csta-emhl.apg.army.mil) 
using  the  format  of  Figure  17-6  to  ATIRS. 

h.  When  the  program  manager  does  not  possess  electronic 
distribution  capability,  the  data  will  be  prepared  in  accordance 
with  the  format  of  Figure  17-6  and  will  be  provided  on  tape, 
floppy  disk  or  other  electronic  storage  media  to  the  ATIRS 
administrator,  who  ensures  input  into  the  database.  No 
hardcopies  will  be  submitted. 

i.  The  program  manager  will  prepare  a  listing,  using 
agreements  reached  by  the  TIWG  members,  for  distribution  of  basic 
corrective  action  data,  photographs,  classified  information  or 
other  information  related  to  a  corrective  action. 

j.  Distribution  of  CA  data  for  tests,  other  than  those 
identified  by  the  TIWG,  is  limited  to  the  addressees  designated 
by  the  program  manager. 

k.  Until  an  automated  support  system  is  established  to 
efficiently  process  picture  and  graphics,  transmission  of 
pictures  and  graphics  by  facsimile  is  encouraged. 

l.  Upon  receiving  the  corrective  action  data,  ATIRS  will 
automatically  assign  a  CA  number  to  block  45  for  tracking  of 
corrective  actions.  This  number  will  be  included  whenever  the 
program  manager  resubmits  corrective  action  data  to  ATIRS  for 
revision. 

17-8.  TIR  Closure  Procedures 

a.  Critical,  major,  and  minor  TIRs  will  initially  be  shown  in 
an  "OPEN”  TIR  status  in  ATIRS.  Information  TIRs  in  ATIRS  will 
automatically  be  in  a  "PENDING”  TIR  status  and  a  CA  status  of 
"OPEN".  TIRs  are  "CLOSED"  upon  receipt  of  a  CA  status  of 
"COMPLETED".  A  "VERIFIED"  CA  status  remains  an  "OPEN"  TIR.  CA 
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statuses  in  ATIRS  are  changed  by  submitting  a  revised  CA  data 
stream  into  ATIRS. 

b.  A  corrective  action  review  team  comprised  of  the  program 
manager,  combat  developer,  operational  evaluator,  and 
developmental  evaluator/assessor  will  review  all  CA  data,  and 
associated  TIRs,  for  proper  CA  categorization.  All  critical, 
major,  and  major-linked  repetitive  minor  TIRs  (see  paragraph 
17-6g  for  further  information  on  repetitive  minor  TIRs)  must  have 
CA  data  entered  into  ATIRS.  Critical  and  major  TIRs  involving  a 
safety  hazard  must  be  coordinated  with  the  safety  community 
before  they  can  be  closed. 

c.  The  corrective  action  review  team  may  be  held  either  in  a 
separate  meeting  or  concurrently  during  any  other  convenient 
meeting  where  corrective  action  statuses  might  be  discussed. 
Telephonic  meetings  are  also  acceptable. 

d.  When  any  member  non-concurs  with  the  proposed  CA  status 
decision,  the  program  manager  will  attempt  to  resolve  the  issue; 
however,  if  it  cannot  be  resolved  he  will  advise  all  members  of 
the  final  decision.  The  member  nonconcurring  may  raise  the  issue 
to  the  next  higher  level  of  management  for  resolution  and  will 
concurrently  advise  the  program  manager.  ATIRS  will  reflect  an 
open  status  until  the  final  decision  by  the  appropriate 
authority. 

e.  When  the  CA  status  is  changed,  the  program  manager  will 
transmit  a  CA  data  stream  to  ATIRS  with  the  CA  status 
information.  CA  status  changes  for  critical,  major,  and 
major-linked  repetitive  minor  TIRs  can  occur  only  after: 

(1)  Appropriate  concurrence  by  the  corrective  action 
review  team. 

(2)  Withdrawal  of  nonconcurrence  or  resolution  by 
intermediate  or  final  decision  authority. 

f.  Timeliness  of  CA  status  change  information.  In  order  to 
affect  continuous  evaluation,  the  program  manager  will  submit  the 
changed  CA  status  information  to  ATIRS  as  soon  as  the  corrective 
action  review  team  has  decided  the  change. 
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TEST  INCIDENT  RE 

PORT  (TIR)  DA  FORM  XXXX-R 

TEST  INCIDENT  REPORT 
(AR  73-1) 

1.  Release  Date:  (MUST  FILL) 

Test  Title: 

2.  (MUST  FILL) 

Test  Project#:  TIR#: 

3.  (MUST  FILL)  4.  (MUST  FILL) 

5.  Test  Agency:  (MUST  FILL] 
7.  System:  (MUST  FILL) 

T  MTV  Trx'D 

6.  Test  Sponsor:  (MUST  FILL) 

8.  (NOT  USED) 

Trptriur  nA'TA  — 

10.  Model:  (MUST  FILL) 

11.  Serial#: 

12.  USA#: 

13.  Mfr: 

14.  Contract#: 

15.  Item#: 

XX  XXT^XT 

Xlbiu  UAIA  ^ 

Test  Life:  Units: 

21. 

22. 

23. 

24. 

Xx  X^wXi 

30.  Title:  (MUST  FILL) 

31.  Subsystem:  (MUST  FILL) 

32.  Incident  Class: 

(MUST  FILL) 

33.  Observed  During: 

34.  Action: 

35.  (NOT  USED) 

X  XJAIA  —  —  —  —  ^  ■ 

40.  Date  &  Time:  (MUST  FILL) 

41.  FD/SC  Step#: 

42.  FD/SC  Class: 

43.  Chargeability : 

44.  Incident  Status 

45.  Corrective  Action# : (RESERVED) 

46.  Categories 

47.  Keywords: 

Test  Environment: 

48. 

49.  Disposition: 

XXX  xxTr«x  rM?XTrn 

Type:  Condition: 

50.  Name:  (MUST  FILL) 

51.  Serial#: 

52.  FSN/NSN: 

53.  Mfr: 

54.  Mfr  Part#: 

55.  Drawing#: 

56.  Quantity:  (MUST  FILL) 

57.  Action:  (MUST  FILL) 

58.  (NOT  USED) 

oUX^Ilid  A" 

60.  FGC: 

61.  LSA#: 

Part  Life  Units: 

62. 

63. 

64. 

65.  Next  Assy: 

66.  Serial  #: 

67.  Software  Version  #: 

m-  iT 


-  IV  MAINTENANCE  DATA  - 

70.  Diagnostic  Clockhours:  80.  Type: 

71.  Diagnostic  Manhours:  81.  Level  Used: 

72.  Total  Maint  Cloc)chburs:  82.  Level  Prsc: 

73.  Total  Maint  Manhours;  83.  Level  Recm; 


V  INCIDENT/MAINTENANCE  DESCRIPTION 
(MUST  FILL  FIRST  LINE) 

(MUST  FILL) 


TIR  Number: 
Number ; 


Page 


(MUST  FILL) 


Name,  Title  &  Phone  of  Preparer: 


FOR  THE  COMMANDER: 


-  VI  CORRECTIVE  ACTION  DATA  - 

CA  Status:  Asgd  Resp:  CA  review  team  date; 

100.  (MUST  FILL)  101.  102.  (MUST  FILL) 


120.  Developer's  Analysis  of  Problem; 

(MUST  FILL) 


121.  Status/Description  of  Corrective  Action: 

(MUST  FILL) 


122.  Test  Results  on  Corrective  Action: 

(MUST  FILL) 


123.  Planned  Production  Implementation; 

(MUST  FILL) 


Figure  17-1. Sample  TIR  Form  2134  &  1901 


Test  incident  report  (TIR)  preparation  instructions 
test  incident  (TI)  data; 


1.  INTRODUCTION. 

a.  This  appendix  contains  the  data  entry  procedures  for  the 
test  incident  data  blocks  in  the  TIR  (DA  Form  XXXX-R)  (i.e. 
excluding  Section  VI  -  Corrective  Action  Data)  and  procedures 
that  should  be  followed  by  test  planning  personnel  prior  to  the 
start  of  test.  The  DA  Form  XXXX-R  is  located  in  Appendix  G. 

b.  Genera"!  policies,  responsibilities,  and  procedures 
pertaining  to  the  use  of  the  TIR  are  contained  in  AR  73-1 
(paragraph  7-12)  and  DA  PAM  73-XX  (basic  pamphlet.  Chapter  12) . 
Test  planning  procedures  are  contained  herein  and  as  notes  in 
paragraph  2,  this  appendix.  Pagination  procedures  and  procedures 
for  augmenting  the  format  of  the  TIR  are  at  paragraphs  3  and  4 , 
this  appendix,  respectively,  following  the  instructions  for 
filling  in  the  DA  Form  XXXX-R. 

c.  When  adding  data  into  ATIRS  using  the  TIR  form,  exact 
placement  and  field  lengths  for  the  data  elements  must  be 
followed  for  a  successful  automated  pickup  of  data.  Appendix  H 
contains  a  template  showing  the  required  placements  and  maximum 
field  lengths. 

d.  Sections  III  and/or  IV  can  be  omitted  if  the  incident  does 
not  involve  a  part/component  or  maintenance  action. 

e.  Some  or  all  of  the  following  materials  for  the  item/system 
under  test  are  required  for  reference  while  preparing  TIRs: 

(1)  System  Support  Package  (SSP)  Component  List. 

(2)  Technical  manuals/equipment  publications. 

(3)  Maintenance  Allocation  Chart  (MAC) . 

(4)  Repair  Parts/Special  Tools  List  (RPSTL) . 

(5)  Logistic  Support  Analysis  (LSA)  Control  Numbers  from 
the  LSA  Record  (LSAR) . 

(6)  Failure  Definition/Scoring  Criteria  (FD/SC) . 

(7)  Technical  Bulletin  750-93-1  (Functional  Grouping 
Codes)  . 

2.  FILLING  IN  SECTIONS  I  TO  V  OF  DA  FORM  XXXX-R. 


Figure  17-2.  TIR  preparation  and  TI  data. 
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a.  Specific  instructions  follow  for  completing  each  area  or 
section  of  DA  Form  XXXX-R. 

b.  If  the  values  on  the  form  are  to  be  typed,  use  either 
10-pitch  or  17-pitch  type.  Do  not  mix  pitch  types;  i.e.,  data  in 
17-pitch  should  not  be  entered  on  a  10-pitch  form. 

c.  Enter  all  data  either  in  numbers,  upper  case  letters,  or 
combinations  thereof,  with  the  exception  of  Section  V,  Incident 
Description,  where  upper-  and  lower-case  characters  are  allowed. 

d.  Do  not'  leave  any  blocks  blank  that  are  designated  "MUST 
FILL.  '• 


e.  Left- justify  all  entries  unless  otherwise  stated  in  the 
instructions. 

TIR  HEADER  AREA. 

Fill  in  the  TIR  header  area  (Blocks  1  through  7)  on  every  TIR 
that  is  prepared. 

BLOCK  1.  Release  Date;  (Cols.  59-78,  X(20)  max) 

Enter  the  date  (in  DD  MMM  YYYY  format)  that  the  TIR  was 
released  for  distribution.  If  a  revised  TIR  is  to  be  issued, 
change  the  original  release  date  to  the  release  date  of  the 
revision,  followed  by  a  space  and  the  phrase  REV#  and  the 
revision  number.  Allocate  two  spaces  for  the  revision  number. 

If  only  one  space  is  used,  fill  in  the  first  space  with  a  0. 

This  is  a  "MUST  FILL" 
block.  Example  follows; 

Original  TIR;  04  AUG  1991 

Revised  TIR;  06  AUG  1991  REV#  01 

BLOCK  2.  Test  Title;  (Cols.  6-39,  X(34)  max) 

Enter  the  title  that  has  been  assigned  to  this  test.  This  is 
a  "MUST  FILL"  block. 

NOTE;  Contact  ATIRS  for  the  test  title  name  prior  to 
commencement  of  testing. 

BLOCK  3.  Test  Project#;  (Cols.  45-60,  X(16)  max) 

Enter  the  test  project  number  that  has  been  assigned  for  this 
test.  This  is  a  "MUST  FILL”  block. 

NOTE;  For  tests  conducted  by  U.S.  Army  Test  and  Evaluation 
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Command  (TECOM)  test  centers,  this  will  be  the  TECOM  Test 
Resource  Management  System  (TRMS)  number,  complete  with  hyphens 
but  without  the  test  center  funding  code  (e.g.,  1-VC-OlO- 
577-011) .  For  tests  conducted  by  non-TECOM  activities,  other 
project  numbers  may  be  applicable.  This  is  a  "MUST  FILL"  block. 

BLOCK  4.  TIR#:  (Cols.  68-77,  X(10)  max) 

Enter  the  TIR  number  that  has  been  assigned  for  this  TIR. 

This  is  a  "MUST  FILL"  block.  Do  not  change  the  TIR  number  (for 
reasons  of  TIR  revisions,  supplementation,  or  whatever)  once  it 
has  been  assigned. 

NOTE:  The  TIR  number  is  made  up  of  two  parts  as  follows: 

a.  The  first  part  (first  4  characters)  is  to  identify  the  TIR 
as  resulting  from  a  specific  test  by  a  specific  tester,  apart 
from  other  tests  by  the  same  or  other  tester  on  a  given  system  or 
model.  The  value  assigned  to  this  part  is  to  remain  constant  for 
the  duration  of  the  test  and  will  consist  of  the  following: 

(1)  The  first  2  positions  are  used  to  identify  the  tester. 
The  value  to  be  assigned  will  be  the  installation  funding  code 
for  the  tester  (if  government)  or  for  the  program  sponsor  (if 
the  test  is  being  conducted  by  a  contractor) . 

(2)  The  third  position  is  to  contain  a  hyphen  (-) . 

(3)  The  fourth  position  is  used  for  a  test  sequence  code 
(values  A  through  Z)  that  relates  to  the  number  of  tests  that 
have  been  performed  by  the  tester  on  a  given  system  or  model 
(e.g.,  assign  "A"  for  the  first  test  of  a  given  system  by  a  given 
tester) . 

b.  An  example  of  the  first  part  entry  for  the  fifth  test  at 
the  U.S.  Army  Combat  Systems  Test  Activity  (CSTA)  on  a  given 
system  is  K2-E.  After  the  alphabet  has  been  exhausted  (excluding 
"I"  and  "0"),  use  the  first  position  from  the  second  part  of  the 
TIR  number  for  additional  codes  (e.g.,  K2-AC) . 

c.  The  second  part  of  the  TIR  number  (up  to  6  characters)  is 
used  for  the  unique  portion  of  the  number.  Normally,  the 
numbering  should  start  with  one  and  be  indexed  by  one  for  each 
TIR;  however,  separate  blocks  of  numbers  may  be  reserved  (e.g., 
for  major  item  types,  individual  end  items,  or  subsystems)  and 
applied  sequentially  when  desired.  Since  this  field  will  be 
sorted  upon,  do  not  allow  any  intermediate  positions  to  be  left 
blank;  also,  require  that  all  numbers  be  right-justified  and 
zero-filled. 

NOTE;  Examples  of  TIR  numbers  are; 
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K2-EA00001 

KC-AOOOIOI 

BLOCK  5.  Test  Agency:  (Cols.  19-38,  X(20)  max) 

Enter  the  name  of  the  test  agency  (government  or  contractor) 
that  is  responsible  for  the  conduct  and  reporting  of  this  test. 
This  is  a  "MUST  FILL”  block. 

NOTE;  Contact  ATIRS  for  the  exact  test  agency  name  prior  to  t 
commencement  of  testing. 

BLOCK  6.  Test  Sponsor:  (Cols.  59-78,  X(20)  max) 

Enter  the  name  of  the  program  sponsor  for  this  test.  The  name 
of  the  program  sponsor  should  include  the  office  symbol  and  test 
sponsor  command  acronym.  This  is  a  "MUST  FILL”  block. 

NOTE:  Contact  ATIRS  for  the  test  sponsor  name  prior  to 

commencement  of  testing. 

BLOCK  7.  System:  (Cols.  14-27,  X(14)  max) 

Enter  the  name  of  the  system  which  encompasses  all  major  items 
to  be  included  in  the  test  program.  This  is  a  "MUST  FILL”  block. 


NOTE:  Contact  ATIRS  for  the  system  name  prior  to  commencement  of 

testing. 

BLOCK  8.  (Not  Used) 

NOTE:  This  block  may  be  used  for  additional  test-unique  or 
commodity-unique  test  program  data,  if  desired.  See  paragraph  4 
of  this  appendix. 

BLOCK  9.  (See  paragraph  4  of  this  appendix.) 

SECTION  I,  MAJOR  ITEM  DATA. 

Complete  this  section  for  every  TIR  that  is  prepared.  With 
the  exception  of  Block  10  and  possibly  Blocks  13  and  14,  specific 
entries  in  the  blocks  are  applicable  only  if  the  TIR  applies  to  a 
single  sample  of  the  major  item  under  test  (e.g.,  an  identifiable 
tank) .  If  the  TIR  is  to  apply  to  more  than  one  sample  of  the 
major  item,  enter  an  appropriate  general  response  (e.g.  ALL,  SEE 
BLOCK  90,  OFF-ITEM,  N/A,  etc.)  in  each  applicable  space  or  leave 
them  blank.  If  "SEE  BLOCK  90”  is  used,  enter  the  appropriate 
values  in  Block  90,  either  in  tabular  or  narrative  form. 

NOTE:  Test  planning  personnel  must  establish  acceptable 
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test-unique  values  for  Blocks  10,  13,  14,  and  15  and  the  units 
for  Blocks  21  through  25,  as  a  minimum,  prior  to  commencement  of 
testing. 

BLOCK  10.  Model:  (Cols.  13-38,  X(26)  max) 

Enter  the  model,  type,  or  series  description  for  the  major 
item  to  which  this  TIR  applies.  This  is  a  "MUST  FILL”  block. 

NOTE:  Contact  ATIRS  for  the  model  name  prior  to  commencement  of 

testing. 

BLOCK  11.  Serial#:  (Cols.  15-38,  X(24)  max) 

Enter  the  major  item  serial  number,  if  applicable.  If  this 
TIR  is  being  used  to  document  an  off-item  repair,  enter  OFF-ITEM 
in  this  space. 

BLOCK  12.  USA#:  (Cols.  17-38,  X(27)  max) 

Enter  the  major  item  USA  registration  number  (or  tail  number) , 
if  applicable. 

BLOCK  13.  Mfr:  (Cols.  11-38,  X(28)  max) 

Enter  the  name  of  the  manufacturer  of  the  major  item,  if 
known . 

NOTE:  Contact  ATIRS  for  the  manufacturer  name  prior  to 

commencement  of  testing. 

BLOCK  14.  Contract#:  (Cols.  17-38,  X(22)  max) 

Enter  the  contract  number,  purchase  order  number,  or  document 
number  that  pertains  to  the  obtainment  of  the  major  item,  if 
known . 

NOTE:  Contact  ATIRS  for  the  contract  number  prior  to 

commencement  of  testing. 

BLOCK  15.  Item#:  (Cols.  13-38,  x(26)  max) 

Enter  the  code  that  has  been  assigned  to  the  end  item,  group 
of  test  items,  or  type  of  data  against  which  this  TIR  is  being 
written. 

NOTE:  This  block  is  to  be  used  by  the  tester  to  assign  test 
unique  codes  in  any  way  he  sees  fit  to  enable  easier  tracking  of 
data.  In  general,  test  planning  personnel  should  establish 
acceptable  test-unique  item  number  codes  prior  to  the  start  of 
test.  Begin  by  determining  whether  all  end  items  to  be  tested 
are  to  be  of  the  same  group  within  the  system  or  of  different 
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groups.  Then  identify  each  end  item  to  be  tested  in  each  group 
and  assign  a  unique  item  number  code  for  each  end  item.  Also 
assign  additional  item  number  codes  for  any  specific  types  of 
data  that  are  to  be  recorded  as  pertaining  to  all  items  within  a 
specific  group  (e.g.  PUBS  for  publication  comments) . 

When  assigning  these  codes,  consider  how  the  test  data  is  desired 
to  be  stored  and  retrieved.  If  data  from  one  or  more  groups  of 
end  items  are  to  be  retrieved  and/or  consolidated,  consider  using 
the  first  character (s)  of  the  code  as  part  of  the  data  retrieval 
selection  criteria. 


BLOCKS  16  to  20. 

BLOCKS  21,  22,  23, 
X(10)  max) 

X(15)  max) 


(See  paragraph  4  of  this  appendix.) 
24  and  25.  Test  Life:  (Cols. 

Units:  (Cols. 


45-54, 

57-71, 


Enter  the  test  life  of  the  major  item  at  the  time  of  the 
•incident  and  its  corresponding  units  of  measure.  Up  to  five 
types  of  major  item  test  life  may  be  entered. 

NOTE:  Contact  ATIRS  for  the  test  life  format  and  units  prior  to 
commencement  of  testing. 


Examples  of  units  of  measure  are  miles,  kilometers,  rounds  , 
fired,  flight  hours,  etc.,  or  abbreviations  thereof.  Test 
planning  personnel  should  assign  a  specific  unit  of  measure  to 
each  block  for  the  duration  of  test,  together  with  required 
spacing,  justification,  and  composition  of  the  test  life  and  unit 
of  measure  entries.  If  a  life  period  other  than  test  life  is  to 
be  recorded,  so  indicate  (e.g.,  TOT  ODOM  MILES). 

BLOCKS  24  to  29.  (See  paragraph  4  of  this  appendix.) 


SECTION  II,  INCIDENT  DATA. 

Complete  this  section  for  every  TIR  that  is  prepared.  The 
blocks  in  this  section  pertain  to  summary  information  and  basic 
incident  data,  to  include  the  various  classifications  of  the  TIR 
and  its  scoring.  Values  entered  in  Blocks  32  and  41  through  43 
should  be  treated  as  preliminary  when  the  TIR  is  first 
prepared.  After  the  TIR  has  been  scored  at  the  RAM  scoring 
conference  or  during  the  TIR  closure  process,  submit  a  revised 
TIR  revising  the  values  entered  in  Blocks  32  and  41  through  43  as 
necessary  to  reflect  the  various  conference  agreements.  The 
status  of  this  scoring  will  be  reflected  in  BLOCK  44. 

NOTE:  Test  planning  personnel  will  establish  acceptable 

test-unique  values  for  Blocks  31,  34,  41,  42,  43,  46,  47  and 
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possibly  48  and/or  49  prior  to  commencement  of  testing. 

BLOCK  30.  Title:  (Cols.  13-38,  X(26)  max) 

Enter  a  title  for  the  TIR  or  a  brief  summary  of  the 
information  that  is  to  be  contained  therein.  This  is  a  "MUST 
FILL”  block.  Be  sure  to  stay  within  the  space  allowed. 

BLOCK  31.  Subsystem:  (Cols.  17-38,  X(22)  max) 

Enter  the  name  of  the  subsystem  to  which  this  TIR  is  to  be 
chargeable.  This  is  a  "MUST  FILL”  block. 

NOTE:  Contact  ATIRS  for  a  list  of  subsystem  names  prior  to 

commencement  of  testing.  The  major  item  name  and  NONE  should 
also  be  included  as  acceptable  values. 

BLOCK  32.  Incident  Class:  (Cols.  22-33,  X(12)  max) 

Enter  the  classification  that  is  to  be  assigned  to  this  TIR. 
Refer  to  DA  PAM  73-XX  (basic  pamphlet,  paragraph  17-6f)  for 
directions  as  to  what  entries  are  applicable.  This  is  a  "MUST 
FILL”  block.  The  only  acceptable  values  are: 

CRITICAL  MAJOR  MINOR  INFORMATION 

BLOCK  33.  Observed  During:  (Cols.  23-38,  X(16)  max) 

Enter  the  word  or  phrase  that  best  describes  the  activity  that 
was  taking  place  when  the  event  occurred  that  prompted  the 
preparation  of  this  TIR. 

NOTE:  Examples  of  typical  test  activity  entries  are: 

INIT  INSPECTION  RAM-D  SAFETY  EVAL 

OPERATION  INSPECTION  NON-MISSION 

MAINTENANCE  TRANSPORT  DESK  AUDIT 

LOG  EVAL  PERF  EVAL  ENV  EVAL 

BLOCK  34.  Action:  (Cols.  14-38,  X(25)  max) 

Enter  the  word  or  phrase  that  best  describes  any  action  that 
was  taken  on  the  major  item  following  the  event  or  incident. 

NOTE:  Contact  ATIRS  for  other  acceptable  values  in  addition  to 

the  below  examples  prior  to  commencement  of  test.  Other  values 
may  be  added  by  registering  them  with  the  ATIRS  administrator. 

Examples  of  entries  for  actions  taken  on  the  major  item  are: 

CLEARED  MAINTAINED  SUSPENDED  TEST 
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n  -i'B 


OPERATED 


DEFERRED  MAINTENANCE 


NONE 


BLOCKS  35  to  39,  These  blocks  may  be  used  for  added  test-unique 
or 

commodity-unique  incident  or  scoring  data,  if  desired.  See 
paragraph  4  of  this  appendix. 


BLOCK  40.  Date  &  Time:  (Cols.  58-77,  X(20)  max) 

Enter  the  date  and  time  when  the  event  occurred  that  prompted 
the  preparation  of  this  TIR.  In  the  case  of  a  TIR  reporting  a 
failure,  malfunction,  discrepancy,  defect,  maintenance  task,  or 
hazard,  this  will  be  the  date  and  time  that  the  problem  or  event 
occurred,  began,  or  was  detected.  For  other  TIRs,  this  will  be 
the  date  and  time  associated  with  determination  of  the  need  for 
the  TIR,  assuming  that  the  requisite  information  is  available. 
This  is  a  "MUST  FILL"  block.  Format  for  entry  is  day,  month, 
year  (DD  MMM  YYYY) ,  24-hour  time,  and  time  standard  (e.g.,  02  AUG 
1991  1315  EST) .  Do  not  attempt  to  list  a  range  of  dates  or 
multiple  dates.  Time  and  time  standard  may  be  omitted,  if  not 
known, 

BLOCK  41.  FD/SC  STEP#:  (Cols.  58-77,  X(20)  max) 

Enter  the  step  number  from  the  FD/SC  decision  tree  flow  chart 
for  the  test  that  best  describes  the  rationale  for  the  scoring  of 
this  TIR. 

BLOCK  42.  FD/SC  Class:  (Cols.  58-77,  X(20)  max) 

Enter  the  FD/SC  classification  that  is  to  be  assigned  to  this 
TIR. 

NOTE:  Contact  ATIRS  for  exact  acceptable  values  prior  to 

commencement  of  test. 

NOTE:  Examples  of  typical  FD/SC  classification  entries  are: 

NO  TEST  NON-RAM  SMA  MAF/MA 

UMA  EMA/UMA  OMF/EMA/UMA 

BLOCK  43.  Chargeability:  (Cols.  60-77,  X(18)  max) 

Enter  the  FD/SC  chargeability  that  is  to  be  assigned  to  this 

TIR. 

NOTE:  Contact  ATIRS  for  exact  acceptable  values  prior  to 

commencement  of  test. 

NOTE:  Examples  of  typical  FD/SC  chargeability  entries  are: 
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h 


HARDWARE 

SOFTWARE 
OPERATOR/ CREW 
MAINT  PERSONNEL 


TRAINING 

PUBLICATIONS 
SUPPORT  EQUIP 
MAINT  HARDWARE 


ENVIRONMENT 

TEST  CONDUCT 

GFE 

NONE 


BLOCKS  44.  Incident  Status:  (Cols.  62-73,  X(12)  max) 

Enter  the  status  that  describes  the  method  of  arriving  at 
values  for  Blocks  32  and  41  through  43.  If  the  tester  scored  the 
data,  enter  PRELIMINARY.  Enter  SCORED  if  a  formal  committee  such 
as  a  RAM  Scoring  Conference  scored  the  data.  Enter  ASSESSED  if  a 
formal  committee  such  a  RAM  Assessment  Conference  scored  the 
data. 

'  > 

Status  entries  are: 

PRELIMINARY  SCORED  ASSESSED 

BLOCK  45.  Corrective  Action#:  RESERVED 

This  is  a  reserved  field.  ATIRS  will  computer  generate  the 
value  for  this  field. 


BLOCK  46.  Categories: 


(Cols.  18-31,  X(14)  max 
33-46,  X{14)  max 
48-61,  X(14)  max 
63-76,  X(14)  max) 


Enter  the  word  or  phrase  from  the  following  list  that  best 
describes  the  categories  or  test  issues  associated  with  this  TIR. 
All  applicable  categories  will  be  submitted,  with  the  primary 
category  listed  first.  Acceptable  values  are  shown  below. 


Acceptable  values  are: 


PHYSICAL 


HUMAN  FACTORS 


CORROSION 


SAFETY 

PERFORMANCE 

RAM 


O&O 

TRAINING 

ENVIRONMENTAL 


TEST  ADMIN 
SOFTWARE 
LOG  SUPPORT 


NOTE:  Contact  ATIRS  for  other  possible  acceptable  categories  in 

addition  to  the  above  listed  prior  to  commencement  of  test. 

Other  values  may  be  added  by  registering  them  with  the  ATIRS 
administrator . 


BLOCK  47.  Keywords: 


Enter  the  word  or  phrase 


(Cols.  18-31, 
33-46, 
48-61, 
63-76, 

of  vital  importance.  All 


X(14)  max 
X(14)  max 
X(14)  max 
X(14)  max) 
applicable 
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I  n  -  ^5“ 


keywords  will  be  submitted,  with  the  primary  keyword  listed 
first. 

NOTE:  Before  using  these  keyword  blocks,  contact  ATIRS  for  a 

list  of  presently  used  keywords  prior  to  commencement  of  test. 
Other  values  may  be  added  by  registering  them  with  the  ATIRS 
administrator. 

BLOCK  48.  Test  Environment:  (Cols.  6~37,  X(32)  max) 

Describe  the  test  environment  that  existed  when  the  event 
occurred  that  prompted  preparation  of  this  TIR.  Use  this  space 
for  information  in  addition  to  that  which  was  entered  in  Block  33 
(Observed  During) .  When  applicable,  cite  the  appropriate 
paragraph  of  a  military  standard  or  specification  in  this  space. 
For  operational  tests,  this  block  normally  contains  the  mission 
profile  that  the  system  was  performing  at  the  time  the  incident 
occur. 

NOTE:  Contact  ATIRS  for  other  acceptable  values  in  addition  to 

those  listed  below  prior  to  commencement  of  test.  Other  values 
may  be  added  by  registering  them  with  the  ATIRS  administrator. 

Examples  of  test  environment  values  are: 

Developmental  Tests: 

AUTOMOTIVE  PERFORMANCE 
ARMAMENT  TEST 
ELECTRICAL  SYSTEM  TEST 
HIGH  TEMPERATURE  CHAMBER 


Operational  Tests: 


MISSION  No:  XXXXXXXX 

Type:  (Cols.  39-60,  X(22)  max) 

Enter  the  environment  type  that  best  describes  the  type 
environment  in  whichthe  test  is  being  conducted. 


NOTE:  Coordinate  with  ATIRS  for  a  list  of  presently  used 

phrases/words  and  to  add  any  other  phrases/words  to  the  list. 


NOTE:  Examples  of  environment  type  values  include: 


PAVED 
GRAVEL 
WASHRACK 
BELGIAN  BLOCK 
FORDING  BASINS 
LABORATORY 


HILLY  CROSS  COUNTRY 
SWAMP/MUD/HOG  WALLOW 
HORIZONTAL  SLOPE 
SIDE  SLOPE 

ENVIRONMENTAL  CHAMBER 
MAINT/REPAIR  SHOP 


VIBRATION 

FUEL  CONSUMPTION 

OBSTACLES 

DYNAMOMETER 

FIRING  RANGE 

NA 
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Condition: 


(Cols.  62-77,  X(16)  max) 


Enter  the  environment  condition  that  best  describes  the 
condition  of  the  environment  in  which  the  test  is  being 
conducted . 


NOTE:  Coordinate  with  ATIRS  for  a  list  of  presently  used 
phrases/words  and  to  add  any  other  phrases/words  to  the  list. 


NOTE:  Examples  of  typical  environment  condition  values  include: 


DRY 

ICE 

WET  SNOW 


DUSTY 
SNOW 
ICE  FOG 


HEAVY  MUD 
LIGHT  MUD 
SAND 


ICE  AND  SNOW 
WET 
NA 


BLOCK  49.  Disposition: 


(Cols.  19-77,  X(59)  max) 


Enter  the  word  or  phrase  that  best  describes  disposition  of 
any  defective  (failed)  materiel  that  pertains  to  this  TIR. 


NOTE:  Contact  ATIRS  for  other  acceptable  values  in  addition  to 
those  listed  below  prior  to  commencement  of  test.  Other  values 
may  be  added  by  registering  them  with  the  ATIRS  administrator. 


Examples  of  typical  disposition  values  include: 


AWAITING  INSTRUCTIONS 
TO  BE  HELD  UNTIL  (date) 

HELD  FOR  FAILURE  ANALYSIS 
TURNED  IN  TO  SUPPLY 

FORWARDED  TO  HIGHER  LEVEL  MAINTENANCE 
RETURNED  TO  (contractor  name) 

RETURNED  TO  (sponsor  name) 

SHIPPED  PER  SPONSOR  INSTRUCTION 


INSTALLED/REINSTALLED 
SCRAPPED 
REWORKED 
CANNIBALIZED 
MISSING/LOST 
OTHER/SEE  BLOCK  90 
NOT  APPLICABLE 


SECTION  III,  INCIDENT  SUBJECT  DATA. 

The  blocks  in  this  section  provide  for  the  description  of  the 
TIR  subject  part  or  assembly  (if  any)  and  its  next  higher 
assembly.  Complete  this  section  if  the  TIR  pertains  in  any  way 
to  an  identifiable  part  or  assembly,  a  major  subassembly  or 
subsystem,  the  major  item  itself,  or  a  component  of  its  SSP.  If 
the  subject  of  the  TIR  is  to  be  a  group  of  parts  or  assemblies  of 
a  given  type,  make  sure  that  all  entries  to  be  made  in  the 
various  blocks  apply  to  the  entire  quantity  that  is  being 
described.  If  the  parts  or  assemblies  in  the  group  have 
different  values  (e.g.,  serial  numbers,  part  numbers,  part  lives, 
etc.),  enter  an  appropriate  general  response  (e.g.,  SEE  BLOCK  90, 
N/A,  etc.)  in  each  applicable  space  or  leave  blank.  Regardless 
of  whether  a  pax't  or  a  group  of  parts  are  of  concern,  provide  in 
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Block  90  a  tabulation  of  the  parts  used.  Detailed  instructions 
are  provided  in  the  Block  90  instructions  below.  Because  Section 
III  contains  summaries  of  data,  its  blocks  should  not  be  used  to 
count  parts  without  close  deliberations. 

BLOCK  50.  Name:  (Cols.  17-38,  X(27)  max) 

Enter  the  name  of  the  part  or  assembly  being  described  as  the 
TIR  subject.  Obtain  it  from  the  RPSTL.  This  is  a  "MUST  FILL" 
block  if  Section  III  is  used. 

BLOCK  51.  Serial#:  (Cols.  15-38,  X(24)  max) 

Enter  the  serial  number,  lot  number,  or  batch  number  for  the 
item  named  in  Block  50. 

BLOCK  52.  FSN/NSN:  (Cols.  15-38,  X(24)  max) 

Enter  the  Federal/National  Stock  Number  for  the  item  named  in 
Block  50.  Obtain  it  from  the  RPSTL. 

BLOCK  53.  Mfr:  (Cols.  11-38,  X(28)  max) 

Enter  the  name  of  the  manufacturer  that  built  or  produced  the 
item  named  in  Block  50,  if  known  or  enter  the  Federal  Supply  Code 
of  Manufacturer  (FSCM)  code  from  the  RPSTL.  Abbreviate  as 
required. 

BLOCK  54.  Mfr  Part#:  (Cols.  17-38,  X(22)  max) 

Enter  the  manufacturer's  part  number  for  the  item  named  in 
Block  50.  Obtain  it  from  the  RPSTL  or  from  the  part  or  assembly 
itself. 

BLOCK  55.  Drawing#:  (Cols.  16-38,  X(23)  max) 

Enter  the  drawing  number  for  the  item  named  in  Block  50,  if 
available. 

NOTE:  If  desired,  figure  and  item  number  references  from  the 

appropriate  RPSTL  may  be  entered  in  this  block  in  lieu  of  a 
drawing  number. 

BLOCK  56.  Quantity:  (Cols.  16-25,  X(10)  max) 

Enter  the  quantity  of  the  items  that  have  been  named  in  Block 
50.  Refer  to  the  introductory  instructions  for  Section  III  if 
the  entry  is  to  be  greater  than  one.  The  number  entered  should 
be  right  justified.  This  is  a  'MUST  FILL"  block  if  Section  III 
is  used. 
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BLOCK  57.  Action: 


(Cols.  14-38,  X(25)  max) 


Enter  the  word  or  phrase  that  best  describes  what  was  done  to 
the  part  or  assembly  named  in  Block  50  following  the  event  or 
incident.  Enter  NONE  if  no  action  was  taken.  This  is  a  MUST 
FILL”  block  if  Section  III  is  used. 

NOTE;  Contact  ATIRS  for  other  acceptable  values  in  addition  to 
the  below  examples  prior  to  commencement  of  test.  Other  values 
may  be  added  by  registering  them  with  the  ATIRS  administrator. 


Examples  of  entries  for  actions  taken  on  a  part  or  assembly 
are: 


INSPECTED 

TESTED 

SERVICED 

ADJUSTED 


CHANGED  MISSION  PROFILE 
DIAGNOSED 
OPERATED 
LUBRICATED 


- 

ALIGNED/REPOSITIONED  DISASSEMBLED/ASSEMBLED 

_ _ 


REMOVED 
MODIFIED 

TORQUED/TIGHTENED 
REMOVED/REINSTALLED 
SAMPLED  OIL/FLUID 

SAFETY  WIRED/SECURED 

PAINTED/ CURING/DRYING 


CLEARED 

DRAINED 

FLUSHED 

PURGED 

LOADED 

ADDED 

CHARGED 

SLAVED 

UNLOADED 

CLEANED/WASHED 

HANDLED/JACKED 

NONE 


CALIBRATED 
INSTALLED 
REPLACED 
DISCONNECTED 
REPAIRED 

OVERHAULED 

REBUILT 

BLOCK  58.  (Not  used) 

NOTE:  This  block  may  be  used  for  additional  test-unique  or 

commodity-unique  test  program  data,  if  desired.  See  paragraph  4 
of  this  appendix. 

BLOCK  59.  (See  paragraph  4  of  this  appendix.) 

BLOCK  60.  FGC:  (Cols.  50-77,  X(28)  max) 

Enter  the  Functional  Group  Code  (FGC)  and/or  the  name  of  the 
functional  group  to  which  the  item  named  in  Block  50  belongs. 
Obtain  it  from  the  RPSTL,  MAC,  or  TB  750-93-1. 


BLOCK  61.  LSA#: 


(Cols.  51-77,  X(27)  max) 


Enter  the  LSA  Control  Number  for  the  item  named  in  Block  50, 
if  applicable.  Obtain  it  from  the  LSAR  for  the  system,  if 
available. 


BLOCKS  62,  63,  and  64. 


Part  Life:  (Cols.  45-54,  X(10)  max) 
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n  -2*? 


Units: 


(Cols.  57-71,  X(15)  max) 


Enter  the  true  life,  if  known,  of  the  item  named  in  Block  50 
and  its  corresponding  units  of  measure.  If  true  life  is  unknown, 
enter  test  life.  If  the  part  or  assembly  is  new,  enter  0  (zero) . 
Up  to  3  types  of  part  life  may  be  entered. 

NOTE:  Contact  ATIRS  for  the  part  life  format  and  units  prior  to 
commencement  of  testing. 

Test  planning  personnel  should  either  assign  a  specific  unit 
of  measure  to  each  block  for  the  duration  of  test  (the  same  as 
for  Blocks  21,  22,  23,  24,  or  25)  or  designate  one  or  more  units 
of  measure  to  be  used  with  specific  parts,  assemblies,  or 
subsystems  of  the  major  item  (i.e.,  the  most  appropriate  units). 
Required  spacing,  justification,  and  composition  of  the  part  life 
and  unit  of  measure  entries  should  also  be  assigned.  The  program 
manager  should  provide  part  life  data  if  this  data  is  not  known. 

BLOCK  65.  Next  Assy:  (Cols.  56-77,  X(22)  max) 

Enter  the  name  of  the  next  higher  assembly  to  the  item  named 
in  Block  50.  Obtain  it  from  the  RPSTL.  The  program  manager 
should  provide  this  information  if  the  RPSTL  does  not  exist. 

BLOCK  66.  Serial#:  (Cols.  54-77,  X(24)  max) 

Enter  the  serial  number,  if  applicable,  of  the  item  named  in 
Block  65. 

BLOCK  67.  Software  Version#:  (Cols.  64-77,  X(14)  max) 

Enter  the  computer  configuration  item  name  when  Categories 
(Block  46)  or  Chargeability  (Block  43)  is  SOFTWARE. 

BLOCKS  68  and  69.  (See  paragraph  4  of  this  appendix.) 

SECTION  IV,  MAINTENANCE  DATA. 

This  section  is  used  for  suitunarizing  data  from  all  applicable 
maintenance  tasks  or  actions  that  were  performed  on  the  end  item 
identified  in  Block  10  as  a  result  of  the  event  or  incident  being 
described  on  this  TIR.  Complete  this  section  if  maintenance  was 
performed.  If  maintenance  is  known  to  be  required  but  is  not 
performed  immediately,  complete  this  section  with  all  available 
known  data,  leaving  the  remaining  spaces  blank.  When  the 
maintenance  is  eventually  performed,  revise  and  update  the  data 
in  this  section  and  on  the  remainder  of  the  TIR  to  reflect  the 
additional  information  learned  during  the  maintenance.  Provide 
in  Block  90  a  tabulation  of  the  clock  hours  and  manhours  by 
maintenance  level  and  type.  Detailed  instructions  are  provided 
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in  the  Block  90  instructions  below.  Because  the  blocks  in 
Section  IV  contain  summaries  of  data,  they  should  not  be  used  to 
calculate  supportability  indices  (e.g.  mean  time  to  repair 
(MTTR) ,  maintenance  ratio  (MR),  etc.)  without  close 
deliberations. 

NOTE:  The  tester  establishes  acceptable  test-unique  values  for 

Blocks  80  through  83  through  the  TIWG  process. 

BLOCKS  70  and  71.  Diagnostic  Clockhours/Manhours :  (cols.  31-37, 

X(7)max) 

Enter  the  clockhours  and  manhours  required  to  perform  the 
diagnostic  (fault  location)  portion  of  maintenance  for  all  tasks 
or  actions  described  on  this  TIR,  regardless  of  maintenance 
level.  Data  is  to  be  reported  in  the  format  HHHHtMM. 

BLOCKS  72  and  73.  Total  Maint  Clockhours/Manhours:  (cols. 

31-37,  X(7)max) 

Enter  the  clockhours  and  manhours  required  to  perform  all 
maintenance  for  all  tasks  or  actions  being  described  on  this  TIR, 
regardless  of  maintenance  level.  Include  all  diagnostic  time 
from  Blocks  70  and  71.  Data  is  to  be  reported  in  the  format 
HHHH:MM. 

BLOCKS  74  to  79.  (See  paragraph  4  of  this  appendix.) 

BLOCK  80.  Type:  (Cols.  51-77,  X(27)  max) 

Enter  the  word  or  phrase  that  best  describes  the  type  of 
maintenance  that  was  performed.  Make  sure  that  the  entry  does 
not  conflict  with  any  scoring  entered  in  Blocks  41  through  43. 

NOTE:  Contact  ATIRS  for  range  of  possible  other  values  than 

those  in  the  below  examples  prior  to  commencement  of  test.  Other 
values  may  be  added  by  registering  them  with  the  ATIRS 
administrator. 

Examples  of  entries  for  maintenance  type  are  (try  to  limit  to 
these  listed) : 

UNSCHEDULED  ESTIMATED 

SCHEDULED  SIMULATED 

NO  TEST 

BLOCK  81.  Level  Used:  (Cols.  57-77,  X(21)  max) 

Enter  the  name  of  the  highest  maintenance  level  that  was 
actually  used  to  perform  any  of  the  maintenance  being  described 
on  this  TIR. 
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BLOCK  82.  Level  Prsc: 


(Cols.  57-77,  X(21)  max) 


Enter  the  name  of  the  highest  maintenance  echelon  prescribed 
in  the  MAC  which  should  have  been  used  during  this  incident. 
Stated  another  way,  this  is  the  lowest  maintenance  level  that  is 
prescribed  in  the  MAC  or  technical  manuals  as  being  authorized  to 
perform  all  of  the  maintenance  being  described  on  this  TIR.  If 
no  level  is  prescribed,  enter  NONE  or  UNKNOWN,  as  applicable. 

BLOCK  83.  Level  Recm:  (Cols.  57-77,  X(21)  max) 

Enter  the  name  of  the  maintenance  level  that  the  tester 
recommends  for  this  maintenance,  if  different  from  the  prescribed 
level  entered  in  Block  82. 

NOTE:  Contact  ATIRS  for  possible  other  values  for  blocks  81  to 

83  than  those  in  the  below  examples  prior  to  commencement  of 
test.  Other  values  may  be  added  by  registering  them  with  the 
ATIRS  administrator. 


Examples  of  acceptable  maintenance  level  entries  in 
hierarchical  order  for  Blocks  81  to  83  are: 


For  Non-Aviation  Systems: 

CREW/OPERATOR 
ORGANIZATIONAL 
ORG/DS  ASSIST 
MFR  ORGANIZATIONAL 
DS/ORG  ASSIST 
DIRECT  SUPPORT 

MFR  DIRECT  SUPPORT 
MFR  CONTACT  TEAM 
GENERAL  SUPPORT 

MFR  GENERAL  SUPPORT 
SPECIAL  REPAIR  ACTY 
DEPOT 

MFR/UNKNOWN  LEVEL 


For  Aviation  Systems: 

CREW/OPERATOR 
UNIT  (AVUM) 

AVUM/AVIM  ASSIST 
MFR  AVUM 
AVIM/AVUM  ASSIST 
INTERMEDIATE  (AVIM)/DS 

MFR  AVIM/DS 

MFR  CONTACT  TEAM 

INTERMEDIATE  (AVIM)/GS 

MFR  AVIM/GS 
SPECIAL  REPAIR  ACTY 

MFR/UNKNOWN  LEVEL 


Values  of  NONE  and  UNKNOWN  are  also  acceptable  for  Block  82  but 
should  not  be  used  with  Blocks  81  or  83. 

BLOCKS  84  to  89.  (See  paragraph  4  of  this  appendix.) 

SECTION  V,  INCIDENT/MAINTENANCE  DESCRIPTION. 

Complete  this  section  for  every  TIR  that  is  prepared.  The  use 
of  upper  and  lower-case  letters  in  Block  90  is  permitted  and 
encouraged. 
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Section  V  may  be  totally  free  form,  or  Blocks  91  through  97 
may  be  added.  If  desired,  it  may  be  composed  of  several 
preprogrammed  elements  from  other  data  entry  systems  (e.g.,  short 
narrative,  full  description,  and  tabulated  fillers  and  spaces  for 
maintenance  subtasks  performed,  parts  used,  tools  used,  etc.). 

BLOCK  90: 

First  line:  (Cols.  6-77,  X(72)  max) 

Start  the  first  line  in  Column  6  on  the  same  line  as  the 
number  ”90. ••  -  On  the  remainder  of  the  line,  enter  a  brief  summary 
of  the  incident  that  is  being  described  on  this  TIR.  For 
example,  "TRANSMISSION  CLUTCH  PACK  WORN,  NO  REVERSE,  FAULT 
LOCATION  ONLY"  OR  "Transmission  removed  and  replaced  because  of 
worn  clutch  pack."  Be  sure  to  stay  within  the  space  allowed  on 
the  line.  This  is  a  "MUST  FILL"  line. 

Subsequent  lines:  (Cols.  2-77,  X(76)  max) 

On  subsequent  lines,  fully  describe  the  incident  or  event  and 
any  resultant  maintenance  tasks.  This  is  a  "MUST  FILL"  block. 

Use  as  many  lines  as  are  necessary  and  continuation  sheet (s) ,  if 
required.  Use  complete  sentences  and  proper  paragraph 
structuring,  numbering,  and  indentation.  Enter  table  headings 
and  values  as  required  to  amplify  the  narrative.  Use  footnotes, 
if  applicable.  If  desired,  skip  lines  to  separate  paragraphs, 
space  tables  and  table  headings,  and  isolate  footnotes.  Based 
on  the  testers  facts  provide  information  to  as  many  of  these 
questions  as  possible:  What  happened?  How  did  it  happen?  How 
was  it  discovered?  Where  did  it  happen?  Under  what  conditions 
did  it  happen?  Why  did  it  happen?  What  actions,  if  any,  were 
taken?  Include  additional  description  in  instances  where  entries 
made  in  Sections  I  through  IV  require  further  clarification. 
Include  reasons  and/or  justification  for  incident  classification 
assignments  and  scoring  if  they  are  not  self-explanatory. 

For  TIRs  pertaining  to  an  accident  or  environmental  release, 
describe  any  resultant  injuries  or  property  damage.  Include  the 
word  "safety"  or  "health"  and  a  risk  assessment  code  (e.g.  Cat 
I-A)  per  MIL-STD-882,  if  applicable. 

Whenever  possible,  indicate  if  the  cause  of  the  incident  or 
event  is  improper  design  (e.g.  improper  material,  overstressing, 
interfering  parts,  or  other  design  problems)  or  improper 
manufacture.  Describe  any  positive  actions  or  suggested 
solutions  which  appear  capable  of  correcting  the  problem  or  would 
prevent  future  incidents  of  this  type  from  occurring. 

TIRs  which  report  subtest  results  will  identify  the  name  of 
the  individual  subtest  and  reflect  the  following: 
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a.  Test  Findings:  State  the  test  results.  Discuss  the 
analytical  procedures  used  and  test  measurement  accuracy.  Ensure 
that  only  factual  data  are  contained  in  this  paragraph. 

b.  Technical  Assessment:  Discuss  the  significance  of  the 
test  results.  Describe  what  engineering  judgments  were  made. 
Analyze  the  test  results  relative  to  criteria  compliance. 

Discuss  any  significant  problems,  corrective  actions,  or 
suggested  improvements . 

Reference  any  hard-copy  reports,  sketches,  photographs,  or 
correspondence  containing  classified  information  on  the  incident 
or  event  that  are  being  forwarded  separately.  Do  not  include  any 
classified  information  in  this  block  or,  for  that  matter,  in  any 
other  block  on  the  TIR. 

Revise  or  update  this  description  as  more  information  becomes 
available  or  if  additional  maintenance  tasks  are  performed  as  a 
result  of  the  event  or  incident.  Identify  revised  information 
with  the  heading  on  a  separate  line:  "Revision",  the  date  of  the 
revision,  and  test  life.  Enter  the  name  of  the  person  who  is 
responsible  for  the  revised  information,  if  other  than  shown  in 
Block  98.  The  test  director  is  the  ultimate  responsible  person 
for  any  TIR  changes.  For  each  TIR  revision  involving  changes  to 
data  in  Sections  I  through  IV,  enter  a  brief  description  of  the 
changes  and  the  reason (s)  for  the  changes.  All  original  data  are 
retained  during  TIR  revision  to  insure  data  integrity.  Revisions 
may  (1)  add  data  or  (2)  change  erroneous  data  by  citing  the  old 
and  adding  the  correction. 

Maintenance  Time  Information: 

After  the  description  narratives,  provide  a  tabulation  of 
maintenance  time  information  for  the  maintenance  actions 
performed  as  follows:  maintenance  level/echelon,  maintenance 
type,  clockhours,  and  manhours.  Begin  the  maintenance  time 
information  with  "MAINTENANCE  TIME  BREAKDOWN"  centered  in  the  TIR 
form.  Use  the  following  header  conventions  in  naming  the 
columns : 


Content: 

Header: 

Maximum 

Length 

Beginning 

Position: 

Date  maintenance  started  (YYMMDD) 

DateSt 

6 

2 

Date  maintenance  ended  (YYMMDD) 

Time  started  (4-digit  2400  hour 

DateEd 

6 

9 

ck  format) 

Time  ended  (4-digit  2400  hour 

TmSt 

4 

16 

clock  format) 

TmEd 

4 

21 

Maintenance  level/echelon 
Administrative  and  logistic 

Level 

5 

26 
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delay  hours 

Maintenance  type 

Delay 

6 

32 

4 

39 

Diagnostic  clockhours 

Dgchrs 

6 

44 

Total  maintenance  clockhours 

Tmchrs 

6 

51 

Diagnostic  manhours 

Dgmhrs 

6 

58 

Total  maintenance  manhours 

Tmmhrs 

6 

65 

Applicable  (Y)  or  not  applicable  (N) 

App 

1 

73 

Clockhour  and  manhour  field  lengths 
Blocks  70-73. 

are  as  specified  in 

The  maintenance  level  content  is  to  contain  no  more  than  5 
characters.  The  maintenance  type  content  is  to  contain  no  more 
than  4  characters.  The  characters  allowed  for  these  values  are 
less  than  those  allowed  for  Blocks  80  and  81  because  of  the  use 
of  abbreviations  to  save  space.  The  applicable  time  (App)  is  a 
marker  that  can  be  used  to  denote  which  maintenance  periods  are 
applicable  for  calculating  supportability  indices. 

Use  the  following  abbreviations  for  the  more  common 
maintenance  levels: 

CREW  -  Crew 

ORG  -  Organizational 

DS  -  Direct  Support 

GS  -  General  support 

AVUM  -  Aviation  Unit  Maintenance 

AVIM  -  Aviation  Intermediate  Maintenance 

SRA  -  Special  Repair  Activity 

DEPOT  “  Depot 

CONTR  --  Contractor 

Use  the  following  abbreviations  to  indicate  the  more  common 
maintenance  types: 

NT  -  No  Test 

U  -  Unscheduled  maintenance  action 
S  -  Scheduled  maintenance  action 
EST  -  Estimated  maintenance  action 
SIMU  -  simulated  maintenance  action 

Part  Information: 

After  the  description  narratives,  provide  a  tabulation  of 
parts  used.  Begin  the  part  information  with  "PARTS  REPLACED 
DATA"  centered  in  the  TIR  form.  Provide  the  following  part 
information:  nomenclature;  FGC;  numerical  identification(s)  such 

as  the  serial  number,  FSN/NSN,  or  manufacturer  part  number 
(whichever  is  available  for  the  test  item);  part  life; 
maintenance  level/echelon  prescribed  for  replacement;  quantity; 
and  action  taken  on  the  part.  The  program  manager  will  provide 
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the  part  information  to  the 

tester  if 

information  is 

lacking  to 

complete  the  part  information  on  a  TIR 
conventions  in  naming  the  columns: 

.  Use  the  following  header 

Content : 

Header: 

Maximum 
Length : 

Beginning 

Position: 

Nomenclature 

19 

2 

FGC 

4 

22 

Serial  number 

Serial# 

24 

27 

FSN/NSN 

FSN/NSN 

24 

27 

Manufacturer  number 

MfrPart# 

22 

27 

Part  life 

PartLife 

7 

52 

Maintenance  level/echelon 

Level 

4 

61 

Quantity 

Qty 

4 

67 

Action 

7 

72 

The  number  of  characters  allowed  is  to  be  no  longer  than  those 
specified  for  the  corresponding  blocks  in  Section  III  and, 
depending  on  actual  information  content,  can  be  even  shorter. 

The  nomenclature  content  is  to  contain  no  more  than  19 
characters.  The  FGC  code  is  to  be  only  4  characters  long  (the 
requirement  of  28  spaces  for  Block  60  on  the  FGC  information  is 
to  allow  a  descriptive  name  for  the  functional  group) .  The  units 
for  the  part  life  will  be  the  same  as  in  Block  62. 

BLOCKS  91  through  97:  (See  paragraph  4  of  this  appendix.) 

TIR  SIGNATURE  BLOCK  AREA. 

Fill  in  the  signature  block  area  (Blocks  98  and  99)  on  every 
TIR  that  is  prepared.  Each  signature  block  may  be  three  lines 
maximum,  X(34)  maximum  per  line.  Leave  one  blank  line  between 
the  command  line  and  the  names  of  the  individuals. 

NOTE:  Test  planning  personnel  should  establish  acceptable 
entries  for  some,  if  not  all,  of  the  information  to  be  entered  in 
Blocks  98  and  99  prior  to  commencement  of  testing. 

BLOCK  98.  Name,  Title,  &  Phone  of  Preparer: 

(Cols.  6-39,  X(34)  max) 

Enter  the  name,  title,  and  telephone  number  of  the  person 
responsible  for  the  content  and  validity  of  the  information  in 
this  TIR. 

BLOCK  99.  FOR  THE  COMMANDER: 

(Cols.  45-78,  X(34)  max) 

Enter  the  release  signature  block  as  required  by  the  tester. 


Figure  17-2.  TIR  preparation  and  TI  data. 


n-'ifc 


NOTE:  This  is  the  end  of  the  Test  Incident  Data  Portion  of  the 

TIR.  Do  not  enter  anything  after  Block  99  other  than  a  line  of 
hyphens  and  the  form  identification  phrase. 

3.  PAGINATION  PROCEDURES. 

Page  breaks  are  unnecessary  in  TIRs  that  are  distributed 
electronically,  but  may  be  present  when  hard  copy  distribution  is 
being  made.  The  location  of  the  page  break  is  left  to  the 
discretion  of  the  preparer.  Ideally,  the  page  break  should  not 
leave  a  section  title  on  one  page  and  begin  the  data  on  the  next. 

At  the  desired  page  break,  end  the  page  with  the  following  line: 


(continued  on  next  page) 


I 

I 


Start  each  new  page  with  the  following  header; _ 

TIR  Number:  Page  Number; 


Regardless  of  the  number  of  pages,  always  end  the  TIR  with  the 
signature  blocks  (Blocks  98  and  99) ,  the  row  of  hyphens,  and  the 
form  identification  phrase. 


4.  DA  FORM  XXXX-R  AUGMENTATION  PROCEDURES. 

a.  The  DA  Form  XXXX-R  is  a  sec^enced  set  of  standardized 
record  formats,  each  format  containing  either  predetermined 
fillers  or  a  combination  of  fillers  and  spaces  for  entering  data. 
The  form  may  be  subjected  to  automated  document  processing. 
Successful  processing  by  the  method  being  used  depends  upon  rigid 
adherence  to  the  record  sequence  and  the  use  and  content  of  each 
record  format. 


b.  During  processing,  the  computer  will  look  for  particular 
data  elements  in  specific  locations  on  the  form  as  depicted  by 
the  fillers.  Therefore,  fillers  on  the  DA  Form  XXXX-R  must  not 
be  altered  with  respect  to  location  or  content,  and  the 
locations  and  field  lengths  of  the  blocks  for  entering  data 
should  not  be  changed. 

c.  Limited  provisions  have  been  made  to  allow  for  tailoring 
of  the  DA  Form  XXXX-R  by  test  planning  personnel  to  accommodate 
test-  or  commodity-unique  data  entry  blocks.  The  following 
locations  contain  unused  spaces  that  may  be  used: 
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(1)  In  the  TIR  Header  Area  on  the  line  containing  Block  7, 
Block  8  may  be  used  for  added  test  program  descriptor  data. 
Following  that  line,  a  line  containing  Block  9  may  be  inserted 
for  still  more  test  program  data. 

(2)  In  Section  I,  Blocks  16  through  20  and  26  through  29 
may  be  inserted  for  still  more  major  item  data. 

(3)  In  Section  II  on  the  lines  containing  Blocks  36 
through  39  may  be  used  for  added  test-unique  or  commodity-unique 
incident  or  incident  scoring  data. 

(4)  In  Section  III  following  the  line  containing  Blocks  58 
and  67,  lines  containing  Blocks  59,  68,  and  69  may  be  inserted 
for  added  data  pertaining  to  the  incident  subject.  Block  58  may 
be  used  for  added  test-unique  or  commodity  unique  incident  or 
incident  scoring  data. 

(5)  In  Section  IV  following  the  line  containing  Blocks  73 
and  83,  lines  containing  Blocks  74  through  79  and  84  through  89 
may  be  inserted  for  added  types  of  maintenance  data. 

(6)  In  Section  V,  Blocks  91  through  97  may  be  added. 

d.  All  additions  should  be  coordinated  with  the  ATIRS 
administrator  prior  to  the  start  of  test.  Except  for  the  TIR 
Header  Area,  all  added  lines  for  added  data  blocks  should  be 
consecutively  numbered  in  the  left-hand  margin.  New  blocks 
should  be  added  in  pairs  on  a  line-by-line  basis.  Also,  all 
additions  must  be  placed  before  the  TIR  Signature  Block  Area. 

e.  Data  collection  procedures  for  all  test-unique  and 
commodity-unique  additions  should  also  be  established  and 
disseminated  prior  to  start  of  test. 
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Figure  17-3.  Test  incident  report  (TIR)  preparation  instructions 
corrective  action  (CA)  data. 

1.  INTRODUCTION. 

a.  This  appendix  contains  the  data  entry  procedures  for 
Section  VI  of  the  TIR  (DA  Form  XXXX-R)  and  procedures  that  should 
be  followed  by  materiel  proponent  planning  personnel  prior  to  the 
start  of  testing.  The  DA  Form  XXXX— R  is  located  in  Appendix  G. 

b.  General  policies,  responsibilities,  and  procedures 
pertaining  to  the  use  of  the  corrective  action  section  of  the  TIR 
are  contained  in  DA  PAM  73-XX  (basic  pamphlet,  paragraph  17-7). 

2.  FILLING  IN  SECTION  VI  of  DA  FORM  XXXX-R. 

a.  Specific  instructions  follow  for  completing  each  area  or 
section  of  DA  Form  XXXX-R. 

b.  Enter  all  data  either  in  numbers,  upper-case  letters,  or 
combinations  thereof,  with  the  exception  of  Blocks  120-123,  which 
can  be  upper-  and  lower-case  letters. 

c.  Do  not  leave  any  blocks  blank  that  are  designated  "MUST 
FILL” . 


d.  Left- justify  all  entries  unless  otherwise  stated  in  the 
instructions. 

BLOCK  100.  CA  Status:  (MUST  FILL)  (Cols.  7-16,  X(10)  max) 

Enter  OPEN,  VERIFIED,  COMPLETED,  or  PENDING,  indicating  the 
status  of  the  corrective  action  the  program  sponsor  must  take  as 
a  result  of  a  critical  or  major  test  incident.  This  is  a  "MUST 
FILL"  block. 

BLOCK  101.  Asgd  Resp:  (Cols.  31-48,  X(18)  max) 

Enter  the  organization  from  the  following  list  that  best 
describes  the  tester's  knowledge  of  who  should  be  assigned 
responsibility  for  the  corrective  action.  The  only  acceptable 
values  are: 

TESTER  MATERIEL  DEVELOPER  NONE 

TRAINER  COMBAT  DEVELOPER 

OTHER  ALL 

Block  102.  Corrective  action  review  team  date:  (Cols.  56-66, 
X(ll)  max) 

Enter  the  day,  month,  and  year  in  DD  MMM  YYYY  format  that  the 
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corrective  action  review  team  met  to  decide  the  corrective  action 
status  change  in  Block  100.  If  the  TIR  is  closed  without  a 
corrective  action  review  team  (as  provided  with  minor  TIRs  not 
linked  to  a  major  TIR) ,  then  enter  NONE.  If  a  meeting  to  close 
TIRs  did  not  occur,  then  use  the  date  when  final  coordination 
was  achieved  by  other  means  such  as  correspondence  or  phone 
conversations.  This  is  a  "MUST  FILL”  block. 

BLOCKS  120-123.  (Cols.  2-78,  X(77)  max) 

Space  is  provided  for  entering  four  different  types  of 
narratives  that  pertain  to  the  corrective  action.  The  four 
narrative  types,  together  with  their  respective  block  numbers, 
are  as  follows: 

120.  Developer's  Analysis  of  Problem. 

121.  Status/Description  of  Corrective  Action. 

122.  Test  Results  on  Corrective  Action. 

123.  Planned  Production  Implementation. 

Enter  the  block  number  and  the  title  for  the  type  of  narrative 
that  is  being  addressed;  then  prepare  and  enter  the  narrative. 

The  use  of  upper  and  lower-case  letters  is  permitted  and 
encouraged.  Use  complete  sentences  and  proper  paragraph 
structuring,  numbering,  and  indentation.  Enter  table  headings 
and  values  as  required  to  amplify  the  narrative.  Use  footnotes, 
if  applicable.  If  desired,  skip  lines  to  separate  paragraphs, 
space  tables  and  table  headings,  and  isolate  footnotes. 

Use  as  many  lines  as  are  necessary  for  each  narrative  type. 
Complete  one  narrative  and  add  a  line  of  dashes  before  beginning 
another  narrative.  Do  not  attempt  to  continue  a  specific 
narrative  on  another  page  following  a  subsequent  narrative.  Keep 
the  narratives  in  order  by  block  number.  Each  of  the  narratives 
are  "MUST  FILL"  blocks. 

Limit  the  narratives  to  the  corrective  action  and  related 
incident  reports.  Reference  any  hard-copy  reports,  sketches, 
photographs,  or  correspondence  containing  classified  information 
that  are  being  forwarded  separately.  Do  not  include  any 
classified  information  in  the  narratives  or,  for  that  matter,  in 
any  other  blocks. 

Revise  or  update  the  narratives  as  more  information  becomes 
available. 
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Figure  17-4.  Test  incident  report  (TIR) 
field  placements  and  allowable  lengths. 


TEST  INCIDENT  REPORT  (AR  73-1)  1.  Release  Date:  XXXXXXXXX 


Figure  17-4.  TIR  Field  Placements  and  Allowable  Lengths. 


72.  Total  Maint  Clockhours:  XXXX:XX  82.  Level  Prsc:  XXXXX 

73.  Total  Maint  Manhours:  XXXXrXX  83.  Level  Recm:  XXXXX 

- V  INCIDENT/MAINTENANCE  DESCRIPTION - 

9  0 .  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 


TIR  Number:  Page  Number: 

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 

MAINTENANCE  TIME  BREAKDOWN 

DateSt  DateEd  TmSt  TmEd  Level  Delay  Type  Dgchrs  Tmchrs  Dgmhrs 
XXXX  XXXXXX  XXX  XXXX  XXXXX  XXX: XX  XXXX  XXX: XX  XXX: XX  XXX: XX 

Tmchrs  App 
XXX: XX  X 

PARTS  REPLACED  DATA 

Nomenclature  FGC  (Numbering  control  used)  PartLife  Level  Qty 
XXXXXXXXXX  XXXX  XXXXXXXXXXXXXXXXXXXX  XXXXXXXX  XXXX  XXX 

Action 

XXXXX 

Name,  Title  &  Phone  of  Preparer:  FOR  THE  COMMANDER: 

9  8 .  XXXXXXXXXXXXXXXXXX  9  9 . XXXXXXXXXXXXXXXX 

XXXXXXXXXXXXXXXXXX  XXXXXXXXXXXXXXXX 

XXXXXXXXXXXXXXXXXX  XXXXXXXXXXXXXXXX 


- VI  CORRECTIVE  ACTION  DATA  - 

CA  Status:  j  Asgd  Resp:  [  CA  review  team  date: 

100.  XXXXXXXXXX  101.  XXXXXXXXXXXX  102.  xxxxxxxxxxx 

120.  Developer's  Analysis  of  Problem: 

XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX 

121.  Status/Description  of  Corrective  Action: 

XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX- 

122.  Test  Results  on  Corrective  Action: 

XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX 

123.  Planned  Production  Implementation: 

XxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxXxX 
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Figure  17-5.  Test  incident  (TI)  data  stream. 


(To  Be  Determined.  The  data  stream  is  under  development  and  should 
be  available  by  1st  QTR  FY  93.) 
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Figure  17-6.  Corrective  action  (CA)  data  stream. 


(To  Be  Determined.  The  data  stream  is  under  development  and  should 
be  available  by  1st  QTR  FY  93.) 


Figure  17-7.  Army  test  incident  reporting  system  (ATIRS) 
access  application  form. 

(  To  be  provided  in  the  next  DA  staffing.) 
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Chapter  18 

Other  Test  and  Evaluation  Considerations 
18-1.  General 

The  special  considerations  outlined  in  this  chapter  should  be 
addressed  during  the  T&E  of  systems  throughout  their  life  cycle. 
Applicability  of  the  following  considerations  and  scheduling  for 
associated  testing  will  be  addressed  by  the  TIWG  members  and 
included  in  the  TEMP,  TEP,  TDP,  lEP,  TR,  expanded  TR,  TER  and 
lER. 

18-2.  Hazardous  waste  source  reduction 

Under  the  provisions  of  AR  200-1  and  AR  200-2,  consideration 
should  be  given  in  the  early  stages  of  the  acquisition  cycle  to 
reducing  the  use  of  toxic  or  hazardous  materials  when  new  systems 
are  tested. 

18-3.  Climatic  testing 

a.  The  Army  recognizes  four  climatic  design  types;  hot, 
basic,  cold,  and  severe  cold  (AR  70-38).  In  general,  all  Army 
systems  must  operate  in  and  are  designed  (as  a  minimum)  for  the 
basic  climatic  design  type. 

b.  Environmental  testing  requirements  for  military  hardware 
must  be  tailored  to  each  specific  system  according  to  the 
environmental  tailoring  process  described  in  MIL-STD  810  and  AR 
70-38.  The  objective  of  such  tailoring  is  to  ensure  that 
military  equipment  is  designed  and  tested  for  resistance  to  those 
environmental  extremes  that  it  will  encounter  during  its  life 
cycle. 


c.  Appropriate  environmental  chamber  or  laboratory  tests  will 
be  scheduled  early  in  the  acquisition  cycle  to  screen  materials, 
components,  or  entire  items  for  possible  design  materiel  or 
reliability  problems.  Chamber  or  laboratory  test  procedures, 
such  as  those  provided  in  MIL-STD  810  and  MIL-STD  781,  must  be 
tailored  to  ensure  that  the  tests  procedures  incorporate 
realistic  environmental  parameters  and  levels.  For  MIL-STD  781 
type-reliability  testing,  chamber  test  conditions  should  also 
emulate  field-use  conditions  (e.g.,  operational  mode  summary  and 
mission  profile,  workload  software,  and  duty  cycle) .  Chamber 
testing  may  be  used  to  investigate  materials  or  component 
performance  in  conditions  not  expected  to  occur  frequently  under 
natural  conditions.  Requirements  and  timing  for  chamber  and 
natural  environmental  testing  will  be  documented  in  the  T&E 
master  plan. 
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d.  Maximum  use  should  be  made  of  data  from  the  developmental 
and  operational  testing  field-test  sources.  The  tester  must 
ensure  that  field-test  sites  are  selected  to  address 
environmental  challenges  at  appropriate  locations  during 
stress-producing  seasonal  conditions. 

e.  Under  the  basic  climatic  design  type  conditions  defined  by 
AR  70-38  and  MIL-STD  810  and  as  modified  by  the  requirement 
document  and  the  T&E  master  plan,  climatic  tests  (chamber, 
laboratory,  or  actual  field  environment)  will  be  completed  before 
type-classification  standard.  Systems  specifically  designated 
for  use  in  extreme  climates  will  complete  tests  in  the  designated 
environments  before  type-classification  standard.  If  the 
independent  evaluators  determine  that  chamber  or  laboratory 
testing  is  not  adequate  to  address  the  issues,  climatic  tests 
that  are  critical  to  the  basic  acceptability  of  the  system  will 
be  conducted  during  developmental  and  operational  testing  at 
natural  field  environmental  test  sites. 

18-4.  Environmental  impact 

a.  For  critical  environmental  concerns,  testing  is  performed 
to  identify  and  quantify  the  emissions,  effluents,  and  wastes 
produced  by  the  system.  (See  AR  200-2.)  Specific  subtests 
designed  to  measure  emissions,  effluents,  and  wastes  may  be 
prescribed  in  the  T&E  master  plan  and  the  TDP.  These 
environmental  quality  characteristics  and  issues  will  be 
evaluated  in  the  lER. 

b.  In  addition,  the  program  manager  may  request  that  the 
developmental  tester  conduct  tests  and  measurements  to  support 
the  life-cycle  environmental  documentation. 

c.  The  program  manager  will  prepare  environmental 
documentation  consistent  with  the  requirements  of  AR  200-2  to 
address  the  environmental  effects  pertaining  to  the  use  and 
operation  of  Systems  through  the  life  cycle.  The  environmental 
documentation  will  be  provided  to  the  developmental  and 
operational  testers  before  testing  commences  at  an  Army  or 
contractor  facility  or  location. 

18-5.  Joint  Test  and  Evaluation  (JT&E) 

a.  The  Office,  Secretary  of  Defense  (OSD)  develops  and 
administers  testing  programs  requiring  two  or  more  services  or 
other  government  agencies'  participation  in  planning,  conducting, 
or  supporting  OSD  requirements. 
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b.  The  JT&E  may  be  oriented  toward  development  and 
acquisition,  tactics,  or  doctrine  but  is  usually  not  acquisition 
oriented.  The  ODCSOPS  (DAMO-FDT)  will  publish  the  annual  call 
for  Army  JT&E  nominations.  OSD-directed  JT&E  normally  are 
initiated  through  nominations  submitted  to  the  OSD  Senior 
Advisory  Council  (SAC) .  HQDA,  TEMA  and  OPTEC  screen  and 
coordinate  proposals  for  OSD-directed  JT&E  and  consolidate  and 
prioritize  nominations  for  the  ODCSOPS  SAC  member.  The  ODCSOPS 
reviews  the  JT&E  nominations  and  presents  a  coordinated  Army 
position  on  test  proposals  as  a  member  of  the  SAC.  OPTEC  has 
management  responsibility  for  JT&E  programs.  OPTEC  is  the 
resourcer  (Army)  for  JT&E,  which  includes  the  feasibility  study 
and  chartered  phases.  Nominations/participants  in  JT&E  must 
notify  OPTEC  and  coordinate  all  resources  early  in  the  JT&E 
process,  to  ensure  that  requirements  for  personnel,  funding, 
equipment  and  units  are  entered  into  the  appropriate  support 
cycle.  The  different  support  cycles  are;  Management  of  Change 
Window  (MOC) ,  Planning  Programming  Budget  System  (PPBS)  and  the 
TSARC.  When  the  Army  is  the  lead  service  for  an  OSD  chartered 
JT&E,  OPTEC  provides  the  initial  administrative  assistance  to  the 
designated  Joint  Test  Director  (JTD) .  DODI  5000.2,  Part  12, 
Section  B  and  DOD  5000.3-M-4,  JTE  Procedures  Manual,  will  be 
utilized  for  all  JT&E.  Additionally,  a  joint  test  and  evaluation 
handbook  (draft)  is  available  for  use  pending  publication  by  OSD. 

18-6.  Multiservice/Joint  Test  and  Evaluation 

Multiservice  T&E  is  acquisition  process  testing.  It  is  a  type  of 
JT&E  conducted  on  a  system  being  acquired  by  more  than  one  DOD 
Component.  Multiservice  T&E  planning,  conduct,  reporting,  and 
evaluation  shall  include  the  participation  and  support  of  all 
TIWG  members.  The  Lead  Service  will  have  overall  responsibility 
for  management  of  the  MOT&E  program  and  will  ensure  that 
Supporting  Service  requirements  are  included  in  formulation  of 
the  basic  resource  and  planning  documents.  The  Supporting 
Service  will  ensure  that  all  of  their  requirements  are  made 
known,  and  will  assist  the  Lead  Service  in  execution  of  the  T&E 
program.  The  lead  service  is  responsible  for  preparing  and 
coordinating  a  single  TEMP,  a  single  test  plan,  and  a  single  test 
and  evaluation  report  reflecting  the  system's  technical 
performance  and  operational  effectiveness  and  suitability  for 
each  service  component.  Coordination  of  the  TEMP  will  be  as 
depicted  in  Part  II,  this  pamphlet.  Multiservice  T&E  are 
normally  initiated  by  a  Joint  Service  Operational  Requirement 
(JSOR)  (See  AR  71-9) .  Testing  procedures  for  multiservice  test 
and  evaluation  follow  those  of  the  lead  service,  with  variance  as 
required,  resolved  through  mutual  agreements. 

18-7.  Limited  Procurement-Urgent  (LPU)  and  Other  Expedited 
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Acquisitions 

a.  LP  type  classification  is  used  when  a  materiel  item  is 
required  for  special  use  for  a  limited  time.  The  specified 
limited  quantity  for  the  LP  item  will  be  procured  without  intent 
of  additional  procurement  of  the  item  under  this  classification. 
The  LPU  type  classification  is  used  to  meet  URGENT  operational 
requirements. that  cannot  be  satisfied  by  an  item  type  classified 
Standard  (STD) . 

b.  Criteria  for  LPU  type  classification  of  an  item  required 
for  urgent  operational  use  will  include  the  following. 

(1)  Existence  of  an  urgent  operational  requirement, 
substantiated  by  the  user  representative  and  combat  developer  or 
by  HQDA. 

(2)  Determination  that  there  is  no  type  classified  item 
that  fully  satisfies  the  requirement. 

(3)  Sufficient  definition  of  the  military  characteristics 
of  the  item  in  materiel  requirements  documents  to  allow 
subsequent  evaluation  of  the  item. 

(4)  Demonstration  that  the  proposed  item  does  not  qualify 
for  STD  and  offers  no  more  than  a  moderate  risk. 

(5)  Determination  that  the  proposed  item  can  be 
economically  maintained  and  logistically  supported  in  the 
geographic  area  and  time  frame  for  which  the  type  classification 
is  valid. 

c.  Type  classification  LPU  will  not  be  used  solely  to  avoid 
test  and  evaluation  of  the  item. 

d.  Not  later  than  six  months  following  delivery  of  the 
initial  shipment  of  the  LPU  item,  the  user  or  requester  of  the 
item  will  collect  data  and  provide  an  operational  field 
evaluation  statement  to  the  program  manager  or  mission  assignee 
agency.  Information  copies  will  be  provided  to  HQDA  (SARD-RPP) , 
TRADOC,  AMSAA,  OPTEC,  and  MRS A. 

18-8.  Acquisition  of  Nondevelopmental  Items  (NDI) 
Nondevelopmental  item  acquisition  is  a  generic  term  that  covers 
systems  or  pieces  of  equipment  which  may  require  limited 
development  effort  by  the  Army.  NDIs  include  materiel  developed 
and  in  use  by  other  U.S.  military  services  or  government 
agencies,  materiel  developed  and  in  use  by  other  countries,  and 
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materiel  available  commercially.  There  are  two  general 
categories  of  NDI  and  a  third  level  of  effort  not  designated  as  a 
separate  category  (AR  70-1) . 

a.  Basic  NDI.  Basic  NDI  consists  of  off-the-shelf  items 
(e.g.  commercial,  foreign,  other  services)  which  will  be  used  in 
the  same  environment  for  which  they  were  designed  and  will 
require  no  modification. 

b.  NDI  Adaption.  NDI  Adaption  consists  of  off-the-shelf 
items  to  be  used  in  an  environment  different  than  that  for  which 
designed  and  which  will  require  modification  to  satisfy 
requirements . 

c.  NDI  Integration.  NDI  Integration  is  focused  on  the 
integration  of  existing  proven  components.  This  category  may 
require  some  hardware  and  software  development  and  integration. 

MS  I  and  MS  II  decisions  occur  very  close  together  and  the 

MS  II  decision  point  is  waived  based  on  a  judgement  that  there  is 
no  need  for  development.  NDI  feasibility  surfaces  during  the 
normal  requirements  generation  process  with  the  preparation  of  an 
Mission  Need  Statement  (MNS)  and  a  preliminary  determination  of 
whether  NDI  is  a  viable  option.  This  determination  by  the  MATDEV 
is  based  on  an  initial  analysis  of  the  operational  requirements 
in  the  MNS  versus  technology  or  materiel  already  developed  and  in 
existence  (e.g.,  foreign-made  materiel).  The  criteria  for  a 
viable  option  is  that  a  facsimile  system  or  elements  of  a  system 
are  already  operationally  successful  and  are  adaptable  to  the 
operational  requirements  specified  in  the  MNS. 

d.  An  important  advantage  of  NDI  alternatives  is  reduced 
acquisition  time.  This  is  accomplished,  in  part,  by  minimizing 
Army  testing  on  NDI.  General  guidance  is  that  we  will  not  test 
when  existing  date  (contractor  or  other  sources)  provides  us 
reasonable  answers.  It  is  imperative  that  independent  evaluators 
get  involved  early,  participate  in  the  NDI  program,  and  provide 
operational  assessments  or  test  and  evaluation  reports. 

e.  Test  and  Evaluation.  No  acquisition,  including  NDI,  is 
exempt  from  DT/OT&E  necessary  to  verify  the  MANPRINT,  quality, 
safety,  reliability,  performance,  supportability, 
transportability,  and  availability  characteristics  of  a  system. 
Tests  by  manufacturers  and  contractors,  previous  performance 
data,  and  market  analysis  information  may  validate  the 
acceptability  of  these  characteristics  and  provide  evidence  of 
system  operational  effectiveness  and  suitability.  However,  for 
the  production  decision  at  least  minimal  OT&E  is  required.  This 
must  follow  standard  DOD  policy  with  regard  to  system  contractor 
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involvement . 

f.  Testing  prior  to  initial  milestone  decision  review. 

Testing  should  be  limited  to  that  absolutely  necessary  to  obtain 
data  necessary  to  make  the  NDI  decision.  This  is  accomplished 
through  the  Market  Investigation  (MI) . 

(1)  As  an  initial  step,  the  Army  will  minimize  testing  by: 

(a)  Obtaining  and  assessing  contractor  test  results. 

(b)  Obtaining  usage/failure  data  from  other  customers. 

(c)  Observing  contractor  testing. 

(d)  Obtaining  test  results  from  independent  test 
organizations,  e.g..  Underwriter's  Laboratory,  National  Bureau  of 
Standards . 

(2)  Initial  data  collection.  If,  based  on  this  initial 
data  collection,  it  is  decided  that  more  information  is  needed  to 
make  a  sound  NDI  decision,  the  MI  may  enter  into  an  evaluation 
phase.  NDI  candidates  may  be  bought  or  leased,  and  developmental 
and  operational  tests  (including  RAM  and  logistic  support)  should 
be  conducted  in  the  operational/combat  environment.  Safety 
procedures,  in  accordance  with  AR  385-16,  must  be  followed  prior 
to  conducting  operational  or  other  user  testing.  The  results 
will: 


(a)  Directly  support  the  AS  to  accept  or  reject  the  NDI 
alternative. 

(b)  Influence  preparation  of  requirements  documents, 
such  as.  Operational  Requirements  Document  (ORD) ,  Training 
Device  Requirement  (TDR) ,  Commercial  Training  Device  Requirement 
(CTDR)  . 

(c)  Assist  in  preparation  of  solicitation  documents. 

(3)  The  test  results  will  not  be  used  to  select  a  specific 
contractor  or  product. 

g.  Testing  after  the  initial  milestone  decision  review.  All 
testing  after  the  initial  milestone  decision  review  must  be 
described  and  justified  in  the  DCP  and  TEMP  and  specifically 
approved  by  the  program  decision  authority. 

(1)  Developmental  Test.  No  DT  will  be  conducted  unless 
U.S.  Army  Materiel  Command  (AMC)  identifies  specific  information 
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needs  that  cannot  be  satisfied  by  contractor  or  -other  test  data 
sources.  The  Army  will  not  demand  test  data  in  rigid  formats, 
but  it  will  be  flexible  in  accepting  data  that  answers  the 
essential  questions.  Risks  associated  with  hardware/software 
modifications  (NDI  category  B)  and  the  third  level  of  NDI  effort 
involving  integration  of  components  will  be  carefully  considered 
when  determining  test  requirements. 

(2)  Operational  Test.  NDI  does  not  automatically  mean  no 
EUTE,  LUT,  lOTE  and  FOTE  will  be  required.  If  AMC  demonstrates 
through  the  Market  Investigation  (MI)  data  that  NDI  products  will 
satisfy  the  requirements  document.  OT  will  not  be  required  with 
OPTEC  and  TEXCOM  concurrence.  This  determination  must  be 
included  in  the  initial  milestone  decision  review  documentation 
(DCP  and  TEMP)  and  approved  by  the  program  decision  authority. 

For  NDI  category  A,  OT  will  not  normally  be  required.  For  NDI 
category  B,  OT  will  be  required  only  when  critical  issues  in  the 
TEP  have  not  been  addressed.  A  prior  concurrence  of  the 
operational  tester  and  evaluator  is  required  to  eliminate  OT. 

(3)  Testing  by  NDI  Category.  Testing  requirements  will  be 
tailored  to  each  specific  system.  The  following  testing  guidance 
by  NDI  category  is  not  a  rigid  requirement,  but  rather  general 
characteristics  of  testing  activities  appropriate  to  each  NDI 
category.  The  goal  of  minimum  testing  still  remains  regardless 
of  NDI  category. 

(a)  Category  A.  No  testing  prior  to  Production 
Qualification  Test  (PQT)  except  when  the  contract  awarded  to  a 
contractor  who  has  not  previously  produced  an  acceptable  finished 
product  and  the  item  is  assessed  as  high  risk.  In  that  case, 
preproduction  testing  should  be  required. 

(b)  Category  B.  Feasibility  testing  is  required  in  the 
military  environment.  Preproduction  Qualification  Test  (PPQT)  is 
required  if  feasibility  testing  results  in  fixes  to  the  item. 

PQT  is  required.  Limited  opferational  evaluation  may  occur  during 
feasibility  and/or  preproduction  tests. 

(c)  Third  level  of  NDI  effort  (Integration  of 
Components) .  Feasibility  testing  is  required  in  military 
environment.  PPQT  of  complete  system  is  required.  Hardware  and 
computer  software  integration  tests  required.  Operational 
testing  required.  PQT  required. 

(4)  Follow-On  Evaluation.  Testing  of  the  NDI  after  the 
first  unit  is  equipped  is  oriented  to  the  validation  and 
refinement  of  operating  and  support  cost/data,  RAM 
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characteristics,  logistic  support,  training,  provisioning 
planning,  etc.  These  tests  can  materially  aid  the  logisticians 
in  supporting  NDI  throughout  its  life-cycle. 

18-9.  Materiel  Change  (MC)  Test  and  Evaluation 

a.  Test  and  evaluation  will  be  conducted  on  all  materiel 
changes  (for  materiel  systems)  and  Post  Deployment  Software 
Support  (PDSS) ,  Information  Systems  (ISs) .  The  scope  and  type  of 
T&E  will  vary  for  each  MC/PDSS  and  will  be  documented  in 
accordance  with  this  pamphlet. 

b.  Test  and  evaluation  of  Materiel  Change  and  Post  Deployment 
Software  (PDS)  require  both  developmental  and  operational 
testing.  The  purpose  of  this  testing  is  to  determine  the 
viability  and  adequacy  of  the  change  and  to  determine  if  the 
change  was  achieved  without  degradation  to  the  system,  other 
components,  or  interface  equipment.  MC  involves  reconfiguration 
of  a  fielded  item  to  provide  new  or  enhanced  capability,  extend 
the  useful  life  of  the  system,  improve  safety  or  readiness,  or 
reduce  operation  support  cost.  MC  includes  Product  Improvement 
Programs  (PIPs)  which  are  not  preplanned. 

c.  MC  is  normally  necessitated  because  of  a  development  (e.g, 
technological  breakthrough,  doctrinal  change,  revised  threat 
estimate)  has  occurred  after  the  original  system  configuration 
was  fixed.  Documentation  and  testing  are  reduced  from  new  system 
development,  as  are  milestone  reviews,  because  both  documents  and 
decision  issues  are  oriented  upon  the  areas  of  change.  The 
general  rule  for  materiel  improvement  applies,  that  if  the 
performance  envelope  changes  significantly,  then  requirements 
documents  are  required,  and  the  associated  TEMP  and  OT&E  cycles 
apply.  However,  if  the  materiel  change  is  focused  only  on 
reduced  operating  costs,  minimal  operational  testing  is  performed 
to  ensure  no  degradation  of  overall  system  performance.  The  lOE 
must  draw  upon  military  expertise,  materiel  acquisition 
knowledge,  and  current  Army  policy  to  make  a  recommendation  to 
the  TIWG.  The  evaluator,  in  coordination  with  the  ODCSOPS  staff 
officer,  informs  DOTE  of  the  materiel  change  if  the  system  has 
been  designated  for  DOTE  oversight. 

d.  The  following  criteria  will  be  used  to  determine  the  T&E 
conduct  on  MC/PDSS: 

(1)  Upon  initiation  of  an  MC  or  PDSS,  the  program  manager 
will  determine  if  formal  coordination  with  the  combat  developer 
or  functional  proponent  is  required.  When  formal  coordination  is 
required  and  the  CBTDEV  or  FP  indicates  that  there  are  critical 
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operational  issues  associated  with  the  MC  or  PDSS,  the  program 
manager  will  charter  a  TIWG.  The  TIWG  will  deteirmine  the  type 
and  amount  of  testing  that  is  required  to  be  performed  on  the 
MC/PDS.  This  testing  will  be  documented  in  the  TEMP.  The  level 
of  program  management  will  be  categorized  based  on  the  same 
criteria  used  for  new  starts,  and  the  decision  authority  or  TEMP 
approval  process  will  be  consistent  with  the  categorization.  An 
OA  or  AOA  will  be  prepared  by  the  developmental  evaluator  or 
assessor,  and  a  developmental  evaluation  or  assessment  report 
will  be  prepared  to  support  the  decision  review  for  approval  of 
procurement  and  application. 

(2)  When  the  formal  coordination  with  the  combat  developer 
or  functional  proponent  is  not  necessary,  or  when  the  combat 
developer  or  functional  proponent  indicates  that  there  are  no 
critical  operational  issues  associated  with  the  MC  or  PDSS,  T&E 
will  be  conducted  as  appropriate,  documented  and  included  in  the 
program  manager's  package  provided  at  the  decision  review.  A 
TIWG,  a  TEMP  and  an  operational  evaluation  are  not  required  in 
this  case.  However,  the  program  manager  will  consult  the  MATDEV, 
Independent  Developmental  Evaluator  or  Assessor,  to  determine  if 
coordination  is  required  with  the  independent  developmental 
evaluator  or  assessor.  When  coordination  is  required,  an 
independent  developmental  evaluation  or  assessment  will  be 
performed  unless  waived  by  the  independent  developmental 
evaluator  or  assessor.  Waiver  of  the  requirement  for  an 
developmental  evaluation  or  assessment  may  occur  if  the 
checksheet  indicates  that  coordination  is  not  required  or  if  the 
developmental  evaluator/assessor,  after  review  of  the  package, 
determines  that  an  evaluation  or  assessment  is  not  needed.  In 
this  case,  formal  notification  to  the  program  manager  will  be 
made  by  the  independent  developmental  evaluator/assessor. 

e.  When  a  block  modification  is  being  tested  or  when  testing 
for  more  than  one  MC  or  type  of  PDS  is  actively  being  conducted 
on  a  system,  and  are  planned  to  be  integrated  into  the  same  end 
item,  efforts  will  be  integrated  so  that  maximum  T&E  efficiencies 
are  achieved.  One  TEMP  (in  those  cases  where  a  TEMP  is  required) 
shall  be  prepared  integrating  the  T&E  required  into  one 
comprehensive  master  plan.  The  TIWG  members  shall  plan  and 
conduct  T&E  so  that  maximum  efficiencies  in  assessing  specific 
aspects  of  the  various  MC/PDSS  can  be  achieved.  In  cases  where 
integration  is  not  practical  (e.g.,  different  program  manager 
agencies) ,  every  effort  shall  be  made  to  assure  that  system 
integration  testing  of  all  MC/PDSS  is  performed. 

18-10.  Army  Clothing  and  Individual  Equipment. 
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Clothing  and  individual  equipment  (CIE)  will  be  tested  and 
evaluated  in  accordance  with  AR  700-86.  CIE  programs  are 
initiated  by  an  approved  Statement  of  Need-Clothing  and 
Individual  Equipment  (SN-CIE)  (AR  700-86).  CIE  are  subject  to  an 
Army  Clothing  and  Equipment  Board  (ACEB)  review,  which 
constitutes  the  IPR  for  those  items. 

18-11.  Test  and  Evaluation  in  Support  of  SOCOM 
The  nature  of  SOCOM 's  mission,  their  requirements  for  peculiar 
equipment  and  the  expediency  in  which  that  equipment  is  needed 
raise  several  issues  in  test  and  evaluation  support.  SOCOM, 

TECOM  and  OPTEC  must  work  together  to  conduct  developmental  and 
operational  test  and  evaluation  in  accordance  with  the  DOD  5000 
series  directives  and  AR  73-1  and  still  meet  the  requirements  of 
SOCOM. 

18-12.  Training  devices  and  ancillary  equipment 
Provisions  of  this  regulation  apply  to  the  T&E  of  all  ancillary 
equipment  and  components  (for  example,  training  devices,  ground 
support  equipment,  and  field  maintenance  test  sets)  (AR  350-38) . 
For  training  devices,  the  developmental  evaluator  will  also 
address  the  critical  technical  parameters,  performance  levels, 
proficiency,  and  effectiveness.  The  operational  evaluator 
addresses  training  effectiveness  and  utility  of  system  training 
devices. 

18-13.  Aiirworthiness 

Aircraft  systems,  subsystems,  and  allied  equipment  will 
be  tested  and  evaluated  (AR  70-62) . 

18-14.  Department  of  Defense  hazard  classification 
All  explosive  systems  and  explosive  components  must  be 
hazard-classified  to  legally  transport  and  store  them  (MIL-STD 
882).  Classification  actions  frequently  require  testing.  It  is 
critical  that  sufficient  items  are  scheduled  for  use  in  these 
tests  and  that  the  tests  are  completed  before  type 
classification. 

18-15.  Security 

Information  that  is  properly  classified  or  that  is  determined  to 
be  unclassified  but  sensitive  (for  example,  proprietary 
information  and  competition-sensitive  information)  must  be 
safeguarded  throughout  the  life  cycle.  A  comprehensive 
protection  and  technology  control  program  shall  be  established 
for  each  program  to  identify  and  protect  this  information  (see 
DoDI  5000.2,  Part,  Section  5).  Procedures  for  implementing  this 
policy  are  contained  in  AR  380-5  and  AR  530-1. 
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18-16.  Communications  security  equipment 

The  National  Security  Agency  is  responsible  for  testing 
communications  security  equipment  during  the  early  stages  of  the 
life  cycle.  Testing  performed  by  the  National  Security  Agency  on 
communications  security  equipment  during  these  phases  will 
fulfill  the  requirements  of  this  regulation. 

18-17.  Range  safety  data 

Army  weapons,  munitions,  and  lasers  require  the  development  of 
range  devices,  as  well  as  safety  data,  to  ensure  safe  and 
effective  testing,  peacetime  training,  target  practice,  and 
tactical  employment  (AR  385-62  and  AR  385-63).  These  data  must 
be  available  before  operational  testing  with  troops.  It  is 
critical  that  sufficient  ammunition  or  explosive  devices  be 
scheduled  for  use  in  the  development  of  these  data  (AR  385-16) . 
(See  AR  70-1.) 

18-18.  Volunteers  as  subjects  of  research 

a.  HUCs  will  be  established  and  will  function  in  accordance 
with  AR  70-25  at  USAOPTEC,  USAMC,  USAISC,  and  USASDC.  As  an 
alternative  to  establishing  an  in-house  HUC,  commanders  may  use 
the  review  process  provided  by  TSG  Human  Subjects  Research  Review 
Board,  as  stated  in  AR  70-25.  These  HUCs  will  review  appropriate 
DTPS,  TDPS,  or  equivalent  documents  to  determine  the  level  of 
risk. 


b.  The  HUC  will  recommend  that  the  test  plan-approving 
official  either  approve,  approve  with  modifications,  defer  review 
to  higher  authority,  disapprove,  or  exempt  from  further  review. 

c.  If  a  HUC  at  USAOPTEC,  USAMC,  USAISC,  or  USASC  determines 
that  the  level  of  risk  is  greater  than  minimal  and  that  the  test 
would  thereby  require  the  participants  to  be  volunteers,  the  test 
plan  or  equivalent  document  will  be  forwarded  to  the  T&E 
Management  Agency  so  that  HQDA  can  review  it  and  submit  a 
recommendation  to  the  Secretary  of  the  Army  as  to  the  approval  or 
disapproval.  The  Secretary  of  the  Army  will  be  the  decision 
authority  in  these  cases. 

d.  Membership  and  administrative  procedures  for  each  HUC  at 
USAOPTEC,  USAMC,  USAISC,  or  USASDC  will  be  provided  to  (and  will 
be  maintained  on  file  at)  the  T&E  Management  Agency. 

e.  After  a  HUC  has  determined  the  level  of  risk,  AR  70-25 
allows  the  HUC  to  recommend  exemption  from  further  human  use 
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review.  After  a  HUC  at  USAOPTEC,  USAMC,  USAISC,  or  USASDC  has 
made  such  a  recommendation,  further  review  of  the  system  or 
equipment  will  not  be  required  unless  the  program  manager,  the 
developmental  or  the  operational  tester  determines  that  changes 
in  the  system  or  equipment,  test  planning,  or  the  method  of 
employment  warrant  such  further  review. 

f.  If  a  HUC  recommends  an  exemption  from  further  human  use 
review,  the  program  manager  will  include  that  determination  in 
part  II  of  the  T&E  master  plan  for  the  system  and  equipment. 

g.  The  HUC  should  determine  the  level  of  risk  based  on  its 
review  of  the  system  and  equipment  in  the  context  of  the  planned 
testing  (based  on  the  test  planning  documentation) ,  the  safety 
release,  the  doctrine,  tactics,  method  of  employment,  and  the 
test  methods  to  be  followed. 

h.  The  HUC  will  ensure  that  functional  expertise  on  the 
system  and  equipment  and  the  location  and  type  of  testing  are 
obtained,  where  appropriate,  to  assist  it  in  determining  the 
risk. 


18-12 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


Chapter  19 

Instrumentation,  Targets,  and  Threat  Simulators  (ITTS) 


Section  I 

Introduction  to  ITTS 


19-1.  Purpose 

This  chapter  provides  a  guide  for  effective  and  efficient 
planning  for  instrumentation,  targets,  and  threat  simulators 
(ITTS)  to  stress  Army  systems  during  materiel  test  and  evaluation 
(T&E)  and  Force  Development  Test  and  Experimentation  (FDTE)  and 
to  capture  the  appropriate  performance  information/data  for 
analysis  and  reporting. 

19-2 .  Scope 

This  chapter  outlines  the  roles,  responsibilities,  and 
relationships  of  key  activities  involved  in  planning  for, 
managing,  and  using  ITTS  in  support  of  T&E  and  FDTE.  Describes 
planning  factors  and  considerations  to  facilitate  efficient  and 
effective  use  of  ITTS.  It  also  Identifies  key  inventory  and 
capability  accounting  systems  and  describes  procedures  for  asset 
scheduling  and  use  and  provides  formats  and  instructions  for 
preparation  and  processing  of  required  documentation. 

19-3 .  Background 

In  1989,  Defense  Management  Review  (DMR)  Decision  936  and  the 
parallel  Army  Management  Review  (AMR)  focused  on  increasing 
management  and  operational  efficiencies.  It  was  determined  that 
substantial  costs  could  be  saved  by  consolidating  operational  T&E 
capabilities  under  a  single  command  (Operational  Test  and 
Evaluation  Command  [OPTEC]),  and  centralizing  funding  and 
development/  acquisition  management  of  targets,  threat 
simulators,  and  major  instrumentation  under  an  Army  Materiel 
Command  (AMC)  chartered  Project  Manager  (PM).  Thus,  in  October 
1991,  the  Army  officially  established  the  Office  of  the  Project 
Manager  for  Instrumentation,  Targets  and  Threat  Simulators  (PM 
ITTS) .  Sustaining  instrumentation  remained  the  responsibility 
of  the  user  (i.e.,  operational  or  developmental  tester)  under  the 
oversight  of  the  PM  ITTS.  In  July  1992,  PM  ITTS  was  officially 
absorbed  under  the  Simulation,  Training,  and  Instrumentation 
Command  (STRICOM)  which  is  a  newly  established  AMC  Major 
Subordinate  Command  (MSC) .  The  mission  and  functions  of  PM  ITTS 
will  essentially  remain  unchanged. 

19-4.  Definitions 

Definitions  of  key  terms  are  presented  below.  Additional 
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definitions  are  found  in  the  Glossary  to  this  pamphlet. 

a.  Instrumentation.  Instrumentation  is  defined  as 
electromagnetic  (e.g.,  electrical,  electronic,  laser,  radar, 
photosensitive,  etc.)  and  other  equipment  (e.g.,  optical, 
electro-optical,  audio,  mechanical,  automated  information,  etc.) 
used  to  detect,  measure,  record,  telemeter,  process,  or  analyze 
physical  parameters  or  quantities  encountered  in  the  test  and 
evaluation  process.  Instrumentation  may  apply  to  a  system  under 
test  or  to  a  target  or  threat  simulator. 

(1)  Major  Instrumentation.  "Major  instrumentation"  is 
defined  as  that  instrumentation  which  satisfies  joint  service 
requirements,  serves  multiple  Army  commands,  requires  a 
significant  level  of  development  and  integration,  or  has  a  large 
dollar  value  (i.e.,  normally  an  acquisition  cost  of  at  least  $1 
million  per  year  or  a  total  cost  of  $5  million  or  more) .  Major 
Army  instrumentation  acquisition  is  normally  Project  Manager  (PM) 
managed . 

(2)  Sustaining  Instrumentation.  "Sustaining 
instrumentation”  is  defined  as  that  instrumentation  which  is  not 
defined  as  major  and  which  satisfies,  within  a  single  command, 
routine  or  recurring  needs  and  normally  has  a  lower  acquisition 
cost  than  major  instrumentation.  Sustaining  instrumentation  is 
normally  acquired  by  the  requiring  command. 

b.  Targets.  Targets  are  normally  economical,  expendable 
devices  used  for  tracking  and/or  engagement  by  missiles/munitions 
in  support  of  T&E  as  well  as  training  missions.  Drone  targets 
are  air  or  ground  vehicles  converted  to  remote  or  programmable 
control.  Ground  targets  are  intended  to  represent  an  adversary 
ground  vehicle  system  or  ground  based  military  structure.  Aerial 
targets  are  intended  to  represent  adversary  aircraft.  Targets 
may  represent  only  selected  adversary  system  characteristics. 

c.  Threat  Simulator.  Threat  simulator  is  a  generic  term  used 
to  describe  equipment  which  represents  adversary  systems.  A 
threat  simulator  has  one  or  more  characteristics  which,  when 
detected  by  human  senses  or  man-made  sensors,  provide  the 
appearance  of  an  actual  adversary  system  with  a  prescribed  degree 
of  fidelity.  Threat  simulators  are  not  normally  expendable. 

19-5.  Roles 

Many  of  the  roles  and  organizational  relationships  depicted  in 
Figure  19-1  and  outlined  below  can  be  found  in  the 
responsibilities  sections  of  other  regulatory  documents. 

However,  some  of  the  roles  of  key  Department  of  Defense  (DoD) , 
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Headquarters,  Department  of  the  Army  (HQDA) ,  and  Army  components 
are  repeated  here  to  provide  the  reader  with  a  condensed  list  for 
easier  reference.  Personnel  who  plan  for  and  execute  T&E 
responsibilities  need  to  be  aware  that  the  roles  listed  below  are 
directly  or  indirectly  related  to  the  planning  for  and 
acquisition  and/or  use  of  ITTS  to  support  T&E. 

(Insert  Figure  19-1.  The  ITTS  Community) 


a.  The  Department  of  Defense  (DoD)  provides  policy  guidance 
and  program  direction;  exercises  Major  Range  and  Test  Facili  y 
Base  (MRTFB)  oversight;  reviews  program  plans;  allocates/defenas 
program  funding. 


b  The  Defense  Test  and  Evaluation  Steering  Group  (DTESG) , 
chaired  by  the  Director,  Office  of  the  Under  Secretary  of  Defense 
(Acquisition)  (OUSD[A]),  has  the  responsibility  of  resolving 
resourcing  issues,  developing  recommendations  on  policy,  ana 
highlighting  significant  issues  in  the  areas  of  developmental 
T&E,  operational  T&E,  and  test  facilities. 


c.  The  Defense  Intelligence  Agency  (DIA)  and  the  Intelligence 
and  Security  Command  (INSCOM)  provide  the  intelligence  base  for 
the  threat  simulator  and  targets  programs  and  exercise  control 
over  all  intelligence  production  and  analysis. 


d.  The  Tri-Service  Executive  Committee  (EXCOM) ,  supported  by 
the  CROSSBOW-S  Committee,  sets  policy,  provides  guidance,  and 
exercises  management  oversight  of  the  Tri-Service  threat 
simulator  program.  The  EXCOM  approves;  resources  for  the  Five 
Year  Defense  Plan  (FYDP) ,  Integrated  Technical  Evaluation  and 
Analysis  of  Multiple  Sources  (ITEAMS)  validation  reports  at 
Design  Specification  Reviews  (DSR)  and  Initial  Operational 
Capability  (IOC) ,  the  Army  Five  Year  Threat  Simulator  Program, 
and  the  lead  service  for  threat  simulator  development. 


e.  The  Multi-Service  Test  Investment  Review  Committee 
(MSTIRC)  is  a  Tri— Service  activity  subordinate  to  the  Joint 
Commanders  Group  on  T&E  (JCG[T&E]),  a  sub  organization  of  the 
Joint  Logistics  Commanders  (JLC) .  The  MSTIRC  is  chartered  to 
review  the  Services*  planned  test  investments  for  duplication; 
interoperability,  commonality,  potential  joint  development,  and 
suitability  as  candidates  for  OSD  central  funding.  The  MSTIRC 
also  recommends  lead  service  assignments  for  ^oint  developments 
and  candidates  for  OSD  Central  T&E  Investment  Program  (CTEIP) 
funding.  Currently,  the  MSTIRC  is  undergoing  ma^or  changes. 

While  still  unofficial,  the  MSTIRC  will  most  likely  be  Phased  out 
and  be  replaced  by  the  T&E  Reliance  Implementation  Board  (TERIB) . 
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The  TERIB  will  retain  the  functions  of  the  MSTIRC  and  also 
include  oversight  of  Reliance  agreements. 

f.  The  Operational  Test  and  Evaluation  Coordination  Committee 
(OTECC)  is  a  multiservice  organization  chartered  by  the  DOT&E  as 
the  management  and  oversight  body  for  the  Resource  Enhancement 
Program  (REP) .  Its  purpose  is  to  offset  shortfalls  in  high 
priority,  short  term  (less  than  3  years)  operational  test 
investment  requirements  which  have  multi-service  application  and 
enhance  realism  of  service  operational  test  and  evaluation.  The 
OTECC  also  coordinates,  plans,  programs,  and  budgets  operational 
test  investment  requirements,  tracks  available  OT&E  assets,  and 
coordinates  service  OT&E  requirements  for  acquisition  of  foreign 
materiel  to  support  operational  testing. 

g.  The  U.S.  Army  Test  and  Evaluation  Management  Agency 
(TEMA) ,  under  the  operational  control  of  the  Deputy  Under 
Secretary  of  the  Army  for  Operations  Research  (DUSA[OR]), 
develops  and  monitors  Army  major  range  and  test  facility  funding 
policy  and  coordinates  T&E  policy,  resources  and  programmatics 
within  the  Department  of  the  Army  (DA)  and  with  DoD.  TEMA  also 
charters  and  designates  the  chairman  of  Validation  Working  Groups 
(VWG)  for  targets  and  threat  simulators. 

h.  The  Army  Test  and  Evaluation  Committee  (ATEC)  provides  a 
forum  by  which  all  elements  of  the  Army  T&E  community,  acting  as 
a  committee  of  the  whole,  formulate  recommendations  to  the  Army 
senior  leadership  regarding  T&E  policy,  procedures, 
organizations,  and  resources. 

i.  The  Army  Instrumentation,  Targets  and  Threat  Simulators 
General  Officer  Steering  Council  (GOSC)  exercises  program 
management  oversight  and  validates  requirements,  priority 
listings,  and  resource  allocations  for  ITTS  development  and/or 
acquisition.  The  ITTS  GOSC  also  recommends  funding  related 
program  changes  to  OSD  and  approves  major  instrumentation  system 
designations. 

j.  The  DA  General  Officer  Test  Schedule  and  Review  Committee 
(GO  TSARC)  reviews  the  Five  Year  Test  Program  (FYTP)  to  identify 
and  coordinate  ITTS  requirements  in  support  of  scheduled  testing. 

k.  HQ  DA,  Deputy  Chief  of  Staff  for  Operations  and  Plans 
(DCSOPS)  chairs  the  HQDA  ITTS  GOSC,  and,  in  conjunction  with 
TEMA,  coordinates  the  ITTS  programs  within  DA  and  prepares  and 
defends  Program  Objective  Memorandum  (POM)  submissions.  DA 
DCSOPS  also  approves  the  Five  Year  Test  Program  (FYTP) . 
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l.  HQ  DA,  Deputy  Chief  of  Staff  for  Intelligence  (DCSINT) 
designates  Threat  Integration  Staff  Officers  (TISO)  for 
Acquisition  Category  (ACAT)  I,  II,  and  OSD  Oversight  systems  to 
assist  developers  and  the  T&E  community  in  generating  and 
articulating  requirements  for  timely,  consistent,  and  continuous 
threat  support  for  developmental  systems.  The  DCSINT  provides 
developers  and  testers  with  validated  intelligence  data  on  threat 
system  capabilities,  characteristics,  tactics,  and  doctrine.  The 
DCSINT  approves  the  Integrated  Threat  Tactical  Operations  Plan 
(ITTOP) .  The .DCSINT  also  functions  as  the  Army  executing  agent 
for  the  Foreign  Military  Exploitation  (FME)  and  Foreign  Military 
Acquisition  (FMA)  programs  and  provides  an  executive  member  for 
the  HQDA  ITTS  GOSC. 

m.  The  U.S.  Army  Materiel  Command  (AMC)  performs  the  duties 
of  materiel  developer  of  assigned  systems  required  by  the  Army. 
AMC  also  provides  an  executive  member  for  the  HQDA  ITTS  GOSC. 

n.  The  U.S.  Army  Simulation,  Training  and  Instrumentation 
Command  (STRICOM)  is  a  newly  established  AMC  MSC.  STRICOM  is 
charged  with  providing  centralized  management  and  direction  for 
the  RD&A  of  Army  Training  Devices,  Simulators  and  Simulations 
(TDSS) ,  major  instrumentation,  targets  and  threat  simulators,  and 
the  Distributed  Interactive  Simulation  (DIS) .  STRICOM  will 
resolve  issues  concerning  ITTS  which  evolve  from  disagreements 
between  PM  ITTS  and  other  AMC  elements  such  as  AMC  PMs  and  Major 
Subordinate  Commands  (MSC) . 

o.  The  Project  Manager  (PM)  ITTS  is  the  Army’s  single  manager 
the  research,  development,  and  acquisition  of  targets,  threat 

simulators,  and  major  instrumentation.  PM  ITTS  manages:  the  Army 
portion  of  the  CTEIP  and  the  Resource  Enhancement  Program  (REP) ; 
planning,  programming,  budgeting,  justifying,-  and  defending 
mission  related  funding;  development  and/or  acquisition, 
fielding,  and  modernization  of  major  instrumentation  systems, 
targets,  and  threat  simulators;  and  inventory/capability 
accounting  (i.e..  Test  Facilities  [TESTFACS]  Register).  The 
Threat  Simulator  Management  Office  (TSMO)  and  the  Targets 
Management  Office  (TMO)  as  a  part  of  PM  ITTS,  serve  as  materiel 
developers  for  assigned  systems.  Work  with  users  of  ITTS  on  the 
identification  and  def initization  of  requirements  documents. 

p.  The  U.S.  Army  Operational  Test  and  Evaluation  Command 
(OPTEC)  chairs  the  General  Officer  (G.O.)  Test  Schedule  and 
Review  Committee  (TSARC) ;  formulates,  prioritizes  and  executes 
the  User  Test  Instrumentation  Program  (UTIP)  in  support  of 
operational  testing;  identifies  and  documents  major 
instrumentation,  targets  and  threat  simulators  requirements  for 
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development  and/or  acquisition  by  PM  ITTS;  functions  as  the 
combat  developer  (CBTDEV)  for  operational  TfieE  threat  related 
systems;  and  provides  an  executive  member  for  the  HQDA  ITTS  GOSC. 
Additionally,  the  OPTEC  Threat  Support  Activity  (OTSA)  has 
responsibility  for  operation  and  maintenance  of  high  fidelity 
Army  Threat  Simulator  assets  and  publication  of  the  ITTOP. 

q.  The  U.S.  Army  Test  and  Evaluation  Command  (TECOM) 
formulates, , prioritizes  and  executes  the  Test  Technology 
Development  and  Acquisition  Program  (TDAP)  in  support  of 
developmental  testing;  identifies  major  instrumentation,  target, 
and  threat  simulator  requirements  for  development  and/or 
acquisition  by  PM  ITTS;  functions  as  the  CBTDEV  for  developmental 
T&E  threat  related  systems;  identifies  and  manages  research  and 
development  of  non-major/sustaining  instrumentation  programs; 
stores,  maintains,  and  conducts  signature  measurement  for  target 
and  threat  simulator  validation;  and  provides  an  executive  member 
for  the  HQDA  ITTS  GOSC. 

r.  The  Defense  Intelligence  Agency  (DIA)  Missile  and  Space 
Intelligence  Center  (MSIC)  and  the  U.S.  Army  Missile  Command 
(MICOM)  provide  matrix  support  to  PM  ITTS  for  development  and 
acquisition  of  Army  targets  and  threat  simulators. 

s.  Testers  identify  their  respective  test  instrumentation, 
targets,  and  threat  simulator  requirements,  document  and  submit 
new/revised  requirements  for  incorporation  into  their  respective 
command  level  programs,  operate  and  maintain  assigned 
instrumentation  and  target  assets,  and  provide  input  to,  TESTFACS. 

t.  U.S.  Army  Training  and  Doctrine  Command  (TRADOC) ,  U.S. 

Army  Forces  Command  (FORSCOM) ,  and  other  Major  Army  Commands 
(MACOM)  identify  requirements  and  manage  assigned  assets.  TRADOC 
also  provides  an  executive  member  for  the  HQDA  ITTS  GOSC. 

u.  Program  Executive  Officers  (PEO)  and  Program/Project 
Managers  (PM)  formulate  and  coordinate  Test  and  Evaluation  Master 
Plans  (TEMP)  for  their  respective  systems  and  identify  (and  fund, 
if  necessary)  system  unique  instrumentation,  target,  and  threat 
simulator  requirements.  They  provide  to  PM  ITTS  all  ITTS 
requirements  for  government  testing  which  cannot  be  accommodated 
by  existing  resources. 

V.  Test  Integration  Working  Groups  (TIWG) ,  chaired  by  the 
materiel  developer,  coordinate  and  integrate  T&E  planning  and 
participation  of  all  members  of  the  acquisition  team.  TIWGs 
facilitate  use  of  appropriate  test  expertise,  instrumentation, 
targets,  threat  simulators,  facilities,  simulations,  and  models; 
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integrate  test  requirements;  accelerate  the  TEMP  coordination 
process;  coordinate  threat  support;  resolve  cost  and  schedule^ 
problems;  and  provide  a  forum  to  ensure  that  test  and  evaluation 
planning,  execution,  and  reporting  are  coordinated. 


Section  II 
ITTS  Planning 


19-6.  General 

Planning  for  instrumentation,  targets,  and  threat  simulators  to 
support  test  and  evaluation  must  begin  early  in  the  T&E  planning 
cycle  to  ensure  timely  and  adequate  support.  The  planning  is 
iterative,  beginning  with  general  requirements,  evolving  into 
specific  requirements,  and  culminating  with  scheduling  or 
recommendations  for  purchase  or  design,  as  appropriate.  ITTS 
planning  is  based  on  combat  developer  derived  Critical 
Operational  Issues  and  Criteria  (COIC)  which  focus  and  support^ 
milestone  decisions  for  U.S.  system  acquisitions.^  COIC  prescribe 
and  provide  a  consistent  emphasis  on  the  user's  minimum  essential 
effectiveness  and  suitability  expectations  from  the  total  system 
at  the  milestone  decision  for  full  production.  COIC  are 
discussed  in  detail  in  Part  Three. 

19-7.  Acquisition  Program  Management  Documents 
During  the  acquisition  of  Army  materiel  systems,  various 
documents  are  required  which  are  either  indirectly  or  directly 
related  to  the  planning  of  ITTS  support.  They  in  turn  are 
related  as  graphically  depicted  in  Figure  19—2.  Documents  which 
do  not  specifically  address, ITTS  but  discuss  the  threat  related 
to  a  particular  materiel  system  are  discussed  below. 

(INSERT  FIGURE  19-2  HERE) 

a.  Operational  Requirements  Document  (ORD) .  The  ORD  is  used 
as  a  basis  for  baselining  and  to  develop  requirements  for 
contract  specifications.  The  ORD  contains  a  brief  threat 
assessment  statement  that  describes  )cey  threat  elements  related 
to  the  mission  deficiency  being  documented. 

b.  Integrated  Program  Summary  (IPS) .  The  IPS  is  generated  by 
the  Materiel  Developer  (MATDEV) ,  in  coordination  with  the  CBTDEV, 
and  contains  an  assessment  (derived  from  the  System  Threat 
Assessment  Report  [STAR])  which  describes  the  DIA  validated 
threat  with  emphasis  on  interactive  effects  of  the  acquisition 
and  the  threat. 
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c.  Cost  and  Operational  Effectiveness  Analysis  (COEA) . 

Threat  in  support  of  the  COEA  is  based  on  an  updated  System 
Threat  Support  Plan  (STSP) ,  the  STAR,  and  other  baseline 
intelligence  products.  Threat  support  from  outside  TRADOC  is 
coordinated  by  the  Threat  Coordinating  Group  (TCG) . 

Additionally,  the  Threat  Manager  (TM)  maintains  an  accurate  audit 
trail  of  all  threat  used  (particularly  that  developed  and 
approved  specifically  for  the  analysis) .  The  COEA  report 
contains  a  summary  of  threat  provided  in  support  of  the  process. 
Addressed  are  the  full  range  of  threat  and  conditions  under  which 
tasks  must  be  performed  throughout  the  life  cycle  of  the  system, 
beginning  at  IOC.  It  also  provides  quantifiable  near-,  mid-,  and 
far-term  system  threat  Scientific  and  Technical  Intelligence 
(S&TI) ,  and  operational  art,  employment,  and  deployment  data. 

d.  Test  and  Evaluation  Master  Plan  (TEMP) .  The  TEMP 
documents  basic  planning  for  all  life  cycle  T&E  related  to  a 
particular  system  acquisition  (See  Part  Two) .  The  TEMP  contains 
an  enumeration  of  new  instrumentation  requirements  in  Part  V,  T&E 
Resource  Summary.  Part  V  also  contains  a  brief  description  of 
the  post-IOC  threat  and  identifies  by  type,  number,  availability, 
and  required  fidelity,  all  targets,  threat  systems,  and/or  threat 
simulators  required  to  support  testing.  Any  major  shortfalls  are 
highlighted. 

e.  Test  and  Evaluation  Plan  (TEP) .  The  TEP  is  a  coordinated, 
single  operational  T&E  planning  document  which  combines  two 
formerly  distinct  and  separately  developed  plans,  the  lEP  and  the 
TDP  (See  Part  Five) .  The  TEP  defines  the  approved  post-IOC 
threat,  including  capabilities,  typical  means  of  operation,  and 
known  means  of  defeating  the  U.S.  system  under  test.  For  MS 
I/II,  it  contains  a  statement  of  potential  targets, 
countermeasures,  and  opposing  weapons  that  a  single  system  can 
expect  to  encounter  on  the  battlefield.  For  post  MS  I/II,  the 
statement  describes  the  threat  to  the  next  higher  level  system  in 
which  the  tested  system  is  embedded. 

f .  Outline  Test  Plan  (OTP)  (See  AR  15-38) .  The  OTP 
identifies  ITTS,  quantities,  and  associated  costs  necessary  to 
realistically  simulate  the  operational  environment,  consistent 
with  test  objectives.  The  OTP  is  prepared  and  maintained  by  the 
designated  tester. 

g.  Threat  Test  Support  Package  (Threat  TSP)  (See  Chapter  9) . 

A  Threat  TSP  is  prepared  for  each  operational  test.  It  is  based 
on  the  STAR  and  is  shaped  by  requirements  derived  from  the  TEMP, 
TEP,  and  OTP.  Threat  TSP  format  is  contained  in  AR  381-11, 
Appendix  B.  The  Threat  TSP  includes  the  STAR,  as  an  appendix. 
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and  other  tailored,  test  specific  appendices  which  address  test 
threat  scenarios,  limitations  (how  the  threat  to  be  portrayed 
differs  from  that  of  the  scenario) ,  a  weapon  and  target  matrix, 
and  Opposing  Force  (OPFOR)  training  plan  for  force-on-force 
testing,  and  a  detailed  tactical  operations  plan  for  threat 
simulator  support,  derived  from  the  ITTOP.  Additionally,  the 
Threat  TSP  will  include  information  related  to  the  technical 
characteristics  and  limitations  of  the  threat  simulators/targets/ 
surrogates  derived  from  accreditation  reports. 

h.  Target/Threat  Simulator  Accreditation  Report.  An 
accreditation  report  serves  to  document  whether  an  individual 
target  or  threat  simulator  is  suitable  to  represent  a  required 
threat  system  for  a  specific  test.  The  report  defines 
differences  which  cannot  be  accommodated  or  offset  in  test 
planning  and  assesses  their  impact.  The  accreditation  report  is 
prepared  by  the  Threat  Accreditation  Working  Group  (TAWG) .  Its 
completion  is  required  sufficiently  in  advance  of  testing  to 
influence  planning. 

i.  Test  Report  (TR) .  A  TR  serves  as  the  primary  record  of  a 
test.  It  presents  factors  and  conditions  under  which  the  test 
was  conducted,  documents  the  resulting  data  base,  and  presents 
the  data  (quantitative  and  qualitative)  resulting  from  the  test. 
The  tester  will  document  deviations  between  planned  and  actual 
threat  representations  by  targets  and  threat  simulators  during 
the  test  for  consideration  by  the  Data  Analysis  Group  (DAG)  and 
independent  evaluator.  The  test  report  is  prepared  and  approved 
by  the  tester,  following  DAG  authentication  of  data  base  validity 
(See  Parts  Four,  Five,  and  Six) . 

j.  Independent  Evaluation  and  Assessment  Reports.  The 
Independent  Evaluation  Report  and  Independent  Assessment  Report 
document  the  analysis  and  evaluation  resulting  from  all  previous 
testing,  analyses,  studies,  modeling,  and  observations  in  support 
of  a  milestone  decision  review.  It  presents  conclusions,  major 
findings,  analysis  and  supporting  rationale  for  the  conclusions, 
a  position  on  the  future  capability  of  the  system  to  fulfill  the 
approved  requirements  in  the  areas  of  operational  effectiveness 
(to  include  survivability)  and  operational  suitability,  and  the 
program  constraints  and  resulting  impacts  on  the  evaluation. 
Threat  related  analysis,  conclusions,  and  findings  play  a  major 
role  in  quantifying  operational  effectiveness.  The  projected 
post-IOC  threat  is  documented  in  the  reports. 


Section  III 

Instrumentation  Support 
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19-8 .  Support  Planning 

Test  plans  and  data  collection  methods  are  discussed  in  Parts 
Four,  Five,  and  Six.  For  ITTS  support  planning,  inventory  and 
capability  accounting  sources  should  be  used  to  identify 
availability  of  specific  instruments  and  types  of  instruments 
necessary  to  capture  the  required  data.  Because  various 
instruments  have  different  performance  characteristics,  support 
requirements,  and  availability,  this  information,  when  integrated 
with  test  schedule  requirements  and  data  accuracy  requirements, 
will  frequently  drive  how  a  test  must  be  structured. 

19-9.  Instrumentation  Support  Plan 

For  tests  not  covered  by  TOPs  or  other  published  Standard 
Operating  Procedures  (SOP) ,  instrument  support  planning  may  be 
articulated  by  the  tester  and/or  instrumenter  in  an  informal 
Instrumentation  Support  Plan  (ISP) .  For  major  tests  with  widely 
diverse  efforts,  a  separate  ISP  may  be  prepared  for  each  major 
category  of  testing  (e.g.,  fire  control  evaluation,  automotive 
performance,  force-on-  force)  The  potential  content  of  the  ISP 
is  discussed  in  the  following  subparagraphs. 

a.  Support  Concept. 

(1)  Purpose.  This  section  defines  the  test  effort 
supported,  who  will  use  the  plan,  and  how  it  is  to  be  used. 

(2)  Scope.  This  section  defines  the  scope  of  the  ISP.  It 
should  bind  the  instrumentation  effort  to  a  specific  set  of  tests 
or  subtests,  as  well  as  identify  what  aspects  of  the 
instrumentation  effort  are  discussed  or  omitted. 

(3)  Approach.  This  section  describes  the  overall  approach 
to  test  execution.  It  should  describe  the  type  of  test  to  be 
conducted,  the  execution  scenario,  the  classes  of  data  to  be 
collected,  a  general  description  of  the  instrumentation  to  be 
used,  and  the  relationship  between  the  instrumentation  and 
operational  aspects  (normally,  engineering  tests  adjust  to  the 
limitations  and  constraints  of  the  instrumentation,  while 
instrumentation  must  be  transparent  to  the  test  players  in 
operational  testing) . 

(4)  System  Description.  This  section  describes,  in  broad 
terms,  how  the  instrumentation  interfaces  to  the  system  under¬ 
going  test.  To  ensure  clarity,  both  the  test  item  and  the 
instrumentation  should  be  described.  The  instrumentation 
description  should  address  each  system  or  device  mounted  on  or 
attached  to  the  system  under  test,  as  well  as  instrumentation 
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remote  from  the  system  under  test.  All  modifications/alterations 
to  the  system  under  test,  interfaces  to  the  system  under  'test, 
and  development  of  modified  and/or  new  instrumentation  must  be 
described  in  operational  terms. 

b.  Data  Collection. 

(1)  Data  Requirements.  The  ISP  specifies  data  requirements 
as  described  in  the  TEP  or  TDP.  Using  this  information,  the  plan 
should  describe  the  instrumentation  source  of  each  data 
requirement.  All  data  must  support  the  evaluation  of  an  issue  in 
the  analysis  plan. 

(2)  Collection  Considerations. 

(a)  Accuracy.  The  desired  and  acceptable  data 
accuracies  should  be  indicated  when  ever  possible.  Both  are 
necessary  pieces  of  information  because  desired  accuracy  may  not 
be  achievable  under  all  conditions.  In  some  cases  the 
interaction  of  instruments,  weather,  geography,  satellite 
constellations,  and  other  factors  can  cause  data  accuracies  to 
change  during  the  conduct  of  a  test.  Without  this  information, 
the  instrumenters  will  be  unable  to  design  data  collection 
configurations  that  will  ensure  all  data  is  within  usable 

parameters.  .  ^  _ 

Further,  they  will  not  be  able  to  monitor  instrument  performance 
during  testing  to  ensure  the  entire  exercise  was  conducted  within 
acceptable  bounds. 

(b)  Criticality.  The  criticality  of  each  data  element 
is  important  in  establishing  proper  instrumentation  support.  In 
some  cases  every  data  requirement  is  critical  because  the  test 
event  may  be  non-repeatable,  such  as  the  firing  of  a  prototype 
projectile  or  missile.  These  more  critical  data  requirements 
must  be  provided  greater  assurance  of  capture.  This  is 
fj^equently  accomplished  by  redundant  collection  systems 

highly  controlled  test  execution.  In  the  operational  testing 
environment,  a  trade  off  may  be  required  between  operational 
realism  and  data  collection.  In  these  decisions,  data 
criticality  is  often  the  overriding  factor. 

c.  Interface  with  Automatic  Data  Processing  (ADP) .  The 
Instrumentation  Support  Plan  requires  close  and  continuous 
coordination  with  the  Data  Management  and  ADP  portions  of  the  DTP 
to  ensure  compatibility.  This  section  includes  plans  to  provide 
appropriate  interfaces  between  the  instrumentation  system (s)  and 
the  ADP  system (s)  which  will  aid  in  the  streamlined  transfer  of 
data  from  the  instrumentation  equipment  to  the  ADP  systems  (files 


19-11 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92 


(DRAFT) 


or  data  bases) .  These  procedures  should  specify  the  techniques, 
protocols,  and  formats  for  accomplishing  the  transfer  of  data  and 
plans  for  verifying  that  complete  transfer  has  occurred. 

d.  Instrumentation  Requirements.  The  instrumentation 
requirements  for  a  given  test  or  subtest  should  be  articulated  in 
this  section.  It  must  be  developed  by  the  tester  in  conjunction 
with  the  instrumentation  section  organic  to  his  organization. 

(1)  Hardware.  This  section  should  list  the  specific 
instruments  or  instrumentation  systems  that  must  be  scheduled, 
borrowed,  or  procured  to  support  the  test  or  subtest.  Where 
multiple  systems  can  accomplish  the  same  data  collection  task, 
they  should  all  be  listed  and  identified  as  suitable 
alternatives.  Common  instruments  (typically  in  plentiful  supply 
and  common  to  most  test  activities)  need  not  be  listed.  However, 
unusual  quantities  of  common  or  expendable  items  should  be 
included.  Nonexistent  instrumentation  listed  as  a  statement  of 
requirement  should  also  be  included  in  the  requiring  activity's 
program  for  development  and  acquisition. 

(2)  Software.  Test  specific  software  such  as  scenarios  for 
simulators,  stimulators,  and  test  specific  instrument  control 
should  be  identified.  Some  complex  instrumentation  may  involve 
software  which  must  be  developed.  This  software  usually  requires 
long  lead  time  for  development,  is  costly,  and  must  be  validated 
well  before  the  commencement  of  testing  to  ensure  data  elements 
are  collected  as  required  and  extraneous  data  are  not  added.  The 
Instrumentation  Support  Plan  should  address  these  considerations 
and  should  alert  test  planners  to  the  lead  time  and  costs 
involved. 

(3)  Personnel.  Identify  all  special  skills  required  to 
support  the  data  collection  effort.  This  should  include  uniquely 
trained  instrumentation  operators  and  technicians,  as  well  as 
interviewers  and  medical  personnel  when  subjective  data  or  human 
use  protocols  are  involved  in  the  data  collection  effort. 

e.  Other  Support  Requirements.  This  is  the  section  that 
addresses  the  logistic  aspects  of  the  instrumentation  support. 

(1)  Security.  A  security  plan  for  instrumentation  must  be 
developed  to  ensure  adequate  physical  and  communications  security 
is  provided  for  collected  data.  While  the  security  plan  is  not  a 
part  of  this  document,  the  need  and  the  instrumentation  inter¬ 
faces  should  be  identified.  Likewise,  all  collection,  storage, 
and  transport  of  classified  data  must  conform  to  security 
regulations  (e.g.,  AR  380-19)  and  red/black  engineering  criteria. 
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(2)  Mobility.  Instrumentation  mobility  requirements  must 
be  fully  articulated.  Both  intra-  and  inter-range  requirements 
must  be  addressed,  to  include  method  of  movement  and  time 
required  to  accomplish  the  task.  Additionally,  specific 
transportation  requirements,  such  as  prime  movers,  special 
trailers,  and  unique  materiel  handling  equipment,  must  be 
articulated. 

(3)  Facilities.  While  most  tests  are  conducted  on 
facilitized  ranges  with  adequate  power,  telecommunications, 
access,  shelter,  and  support  facilities,  some  tests  require 
facilitization  for  accomplishment.  Specific  range  facilities  for 
the  installation,  operation,  and  support  of  the  test 
instrumentation  should  be  articulated  in  this  section. 

(4)  Support  Services.  Special  support  such  as  unique 
calibration  requirements,  installation  and  maintenance  tools,  and 
contract  services,  should  be  fully  articulated  in  this  section. 

f.  Test  Support.  This  section  describes  methods  for 
implementing  the  instrumentation  planning  during  the  phases 
leading  up  to  a  test.  As  a  minimum,  it  should  include  mile¬ 
stones,  demonstrations,  training,  pilot  test,  and  test  readiness 
reviews. 

(1)  Milestones.  The  Instrumentation  Support  Plan  test 
support  section  should  state  specific  milestones  for 
instrumentation  (e.g.,  target  date  to  investigate  types  of 
equipment,  target  date  to  acquire  instrumentation,  target  date  to 
install  equipment,  and  target  date  to  train  personnel  to  use  the 
equipment) . 

(2)  Demonstrations.  Demonstrations  of  instrumentation 
should  be  conducted  within  the  framework  of  the  larger  Test 
Planning  Milestone  Schedule.  The  target  dates,  extent,  and 
number  of  demonstrations  will  vary  from  system  to  system; 
however,  some  general  guidelines  are  helpful.  A  demonstration 
should  probably  precede  each  test  readiness  review.  Also,  if  a 
new  piece  of  instrumentation  is  introduced  relatively  late  in  the 
pre-test  schedule,  and  after  all  other  instrumentation  has  been 
demonstrated,  it  should  not  be  overlooked.  A  date  should  be  set 
to  demonstrate  this  new  piece.  Finally,  a  request  by  one  of  the 
users  (e.g.,  data  manager,  test  director,  evaluator)  to  see  some 
piece  of  instrumentation  demonstrated  again  should  be  granted  if 
at  all  possible.  In  other  words,  demonstrations  should  be 
performed  at  planned  dates  as  well  as  any  additional  times  deemed 
necessary  prior  to  the  test.  In  early  test  planning  stages, 
instrumentation  may  be  demonstrated  by  itself;  however,  at  some 
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time  before  the  Pilot  Test,  the  instrumentation  must  be 
demonstrated  on  or  in  conjunction  with  the  system  to  be  tested. 

(3)  Training.  Specific  dates  should  be  scheduled  for 
training  personnel  in  the  operations  of  instrumentation.  The 
amount  of  training  will  vary  according  to  the  complexity  and 
amount  of  instrumentation,  but  most  of  the  training  sessions 
should  take  place  before  the  last  demonstration  and  last  test 
readiness  review. 

(4)  Pilot  Test.  By  the  date  of  the  Pilot  Test,  as 
scheduled  in  the  DTP,  instrumentation  should  be  in  full  operation 
with  fully  trained  personnel.  For  the  Pilot  Test,  the 
instrumentation  should  be  in  place,  operated  as  if  running  the 
actual  test,  and  should  generate  real  data  for  examination. 

(5)  Test  Readiness  Reviews  (TRR) .  Test  readiness  reviews 
are  discussed  in  detail  in  Parts  Four  and  Five.  This  section 
highlights  only  those  instrumentation  issues  which  should  be 
addressed  during  the  TRR  process. 

(a)  Initial.  At  the  first  review  (approximately  270 
days  in  advance  of  test) ,  a  preliminary  and  general  review  of 
instrumentation  requirements  and  availability  will  be  conducted. 
The  review  should  include  a  tentative  schedule  of  instrumentation 
availability/readiness  milestones  and  should  outline  the  types 
and  extent  of  instrumentation  capability  required  to  support  the 
test.  If  instrumentation  is  being  built  to  satisfy  test  needs,  a 
status  report  of  development  progress  will  be  reviewed. 

(b)  Intermediate.  At  the  second  review  (approximately 
60  to  90  days  in  advance  of  test) ,  the  instrumentation  schedule 
will  be  updated  and  the  instrumentation  concept  laid  out  in 
detail,  including  additions  and  deletions  to  the  preliminary 
plan.  Further,  any  noticeable  problems  with  any  particular 
instrumentation  will  be  addressed.  Resulting  from  this  review 
will  be  a  decision  concerning  the  requirement  for  and  timing  of 
training  of  test  instrumenters. 

(c)  Final.  At  the  third  (final)  review  (occurs  just 
after  the  Pilot  Test) ,  the  status  of  instrumentation  readiness 
will  be  assessed  based  upon  the  accuracy/ adequacy  of  data 
collected  during  the  Pilot  Test  (or  earlier  demonstrations  or 
trials) .  The  purpose  of  the  assessment  will  be  to  isolate 
problems  and/or  deficiencies  remaining  in  the  ISP  and  to  analyze 
their  impact  on  test  readiness.  The  assessment  of  how  well  each 
instrumentation  system  works  in  conjunction  with  the  other 
instruments  and  how  well  each  instrumentation  system  interfaces 
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with  applicable  ADP  systems  must  also  be  reviewed.  If  problems 
are  surfaced  in  these  areas,  corrective  actions  and  their 
potential  impact  on  test  start  should  be  highlighted.  Another 
result  of  this  review  will  be  a  decision  as  to  whether 
instrumenters  have  been  adequately  trained  to  instrument  the  test 
and  are  aware  of  and  trained  in  contingency  plans  for  problems 
that  may  occur  during  the  test. 

g.  Conduct  of  Test.  During  the  actual  conduct  of  the  test, 
the  major  emphasis  for  instrumentation  is  on  properly  executing 
the  support  plan  and  on  ensuring  the  data  collected  through 
instrumentation  flow  smoothly  into  the  data  management  system. 
Accordingly,  knowledgeable  instrumenters  should  be  present  at  the 
test  site  to  expeditiously  correct  any  problems  whenever 
instrumented  data  is  critical  to  the  overall  data  collection 
procedure . 

19-10.  Needs  Satisfaction 

Direct  test  support  instrumentation  needs  will  normally  be 
satisfied  from  assets  on-hand  within  the  test  directorate  or 
activity  assigned  responsibility  for  the  test.  Satisfaction  of 
needs  in  excess  of  organic  capability  will  make  use  of  one  or 
more  of  the  following  methods,  listed  in  order  of  preference. 

a.  Borrow.  Testers  are  encouraged  to  survey  and  query 
existing  inventory  data  bases  (e.g. ,  OPTEC  Instrumentation  Data 
Base  [OIDB]  and  Test  Facilities  [TESTFACS]  Register),  catalogs, 
etc. ,  to  determine  what  additional  needed  resources  are  in 
inventory,  where,  and  in  what  quantities.  Direct  coordination 
with  documented  points  of  contact  (POC)  is  also  encouraged  for 
the  tester  to  gain  a  complete  understanding  of  an  item's 
capabilities,  limitations,  support  requirements,  suitability, 
etc.,  and  to  determine  its  potential  availability.  Schedule  and 
use  requirements  are  discussed  later  in  this  pamphlet;  however, 
it  should  be  noted  here  that  the  borrowing  of  instrumentation 
will  normally  be  affected  through  Property  Book  Officer  (PBO) 
channels  (for  accountability)  and  that  the  borrower  will,  in  all 
probability,  incur  some  fiscal  liability  (e.g.,  round-trip 
transportation  costs).  Additionally,  the  tester  should  be  aware 
that  borrowed  instrumentation  must  be  returned  in  its  original 
configuration,  unless  otherwise  coordinated  in  advance  with  the 
lender. 

b.  Lease.  Standard  off-the-shelf  instrumentation  (e.g., 
visicorders,  oscilloscopes,  digital  voltmeters,  signal 
generators,  and  others  normally  falling  within  the  sustainment 
instrumentation  category)  may  be  leased  or  rented  to  satisfy 
short  term  inventory  augmentation  or  one-time  needs.  However, 
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total  lease/rental  costs  which  could  be  incurred,  to  include  any 
additional  costs/penalties  in  the  event  of  extension(s) 
necessitated  by  a  test  slippage,  should  be  realistically  compared 
to  Non-Development  Item  (NDI)  procurement  costs  before  this 
option  is  pursued. 

c.  Modif ication/NDI  Procurement.  Testers  will  determine  the 
extent  of  modification  to  an  on-hand  inventory  asset  needed  to 
satisfy  their  requirement.  A  trade-off  analysis  of  modification 
versus  procurement  of  NDI  (assuming  availability)  should  be 
conducted  to  determine  the  most  cost  efficient  approach. 

Normally,  NDI  will  be  the  option  of  choice  if  it  is  more 
responsive/ timely  and  procurement  costs  are  less  than  25%  more 
than  those  for  modification  of  an  on-hand  asset. 

d.  Full  Scale  Development.  Design,  development,  and 
procurement  of  instrumentation  for  direct  test  support  will  be 
the  exception  due  to  the  time  required  (experience  has  shown  that 
the  acquisition  cycle  for  sustaining  instrumentation  can  easily 
take  3-5  years  and  8-12  years  is  not  uncommon  for  a  major 
instrumentation  system) .  When  full  scale  development  of 
instrumentation  is  necessitated,  the  impact  must  be  closely 
coordinated  through  the  TIWG  and  the  TSARC,  documented  first  in 
ORD  format,  by  the  Requirer,  for  processing  as  prescribed  in 
Section  9.4,  and  second  in  the  TEMP,  by  the  TIWG,  as  a  potential 
test  limitation. 


Section  IV 

Target/Threat  Simulator  Support 
19-11.  General 

Army  systems  are  developed  to  defeat  existing  and  potential 
threats.  There  is  therefore  a  need  to  test  against  realistic 
targets  and  simulators  of  those  threats.  The  development  and  use 
of  targets  and  threat  simulators  is,  accordingly,  an  integral 
part  of  the  Army  materiel  acquisition  process  and  necessitates 
planning  sufficiently  in  advance  to  ensure  availability  when 
required. 

a.  Development  Planning.  Development  planning  makes  use  of  a 
long  range  (10-15  year)  focus  geared  to  progressing  from 
definition  of  generic  threat  technological  advancements  (e.g., 
directed  energy)  which  have  potential  impact  on  current  and/or 
future  U.S.  systems,  to  specific  applications  of  technology 
(e.g.,  directed  energy  weapons)  which  directly  threaten  or  are  to 
be  countered  by  U.S.  systems.  This  planning  makes  use  of  an 
interactive,  relational  data  base  (Long  Range  Planning  System 
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[LRPS]  Army  Site  Configuration  [ASC]),  which  draws  initially  from 
the  Army  Science  and  Technology  Master  Plan  (STMP) ,  TRADOC's 
Concept  Based  Requirements  System  (CBRS) ,  National  and  DoD 
intelligence  community  products,  and  system  acquisition 
documentation  to  establish  system  links  (threat  to  and/or  target 
of) .  Planning  then  evolves  into  focusing  S&TI  Centers  and  other 
intelligence  organizations  on  specific  systems  through  initiation 
of  Intelligence  Production  Requirements  (IPR)  and/or  through  the 
use  of  ITEAMS  as  necessary.  Candidate  systems  for  development 
are  identified  by  and/or  to  potential  users  (combat  developers, 
PEO/PM  offices,  evaluators,  and  testers)  for  consideration 
through  direct  coordination,  TIWG  interface,  and  annual 
requirements  conferences  chaired  by  PM  ITTS.  Those  candidates 
for  which  specific  needs  can  be  justified  are  subsequently 
documented  in  ORD  format.  TECOM  functions  as  the  combat 
developer  for  developmental  test  and  evaluation  threat  related 
systems.  OPTEC  functions  similarly  for  operational  test  and 
evaluation  systems. 

b.  Use  Planning.  Use  planning  for  targets  and  threat 
simulators  in  support  of  T&E  is  a  cooperative  effort  between  the 
intelligence,  evaluation,  research  and  development  (R&D) ,  and 
testing  communities:  intelligence  officers  identify  and  describe 
the  system  specific  threat  in  all  its  aspects;  evaluators 
determine  which  threat  sensitive  issues  must  be  addressed  by 
test;  developers  manage  the  development  and  acquisition  of  threat 
representative  assets;  and  testers  schedule  and  control  threat 
representative  assets  in  accordance  with  intelligence 
descriptions  and  estimates.  For  a  myriad  of  reasons,  not  the 
least  of  which  are  system  priority  instability  and  available 
funding,  use  planning  normally  has  a  short  range  (1-3  years) 
focus  and  therefore  normally  results  in  reliance  on  existing 
targets  and  threat  simulators  for  need  satisfaction.  This 
reality  underscores  the  critical  importance  of  long  range 
planning  as  discussed  above. 

19-12.  Related  Documents 

When  planning  for  the  use  of  appropriate  targets  and  threat 
simulators,  it  is  important  to  know  how  information  on  the  threat 
to  a  U.S.  system  is  derived  and  where  the  information  is 
documented.  While  these  documents  support  and  justify  the 
development  of  materiel  systems,  they  are  also  used  to  identify 
targets  and  threat  simulators  required  for  T&E  of  the  system. 

Some  of  the  key  threat  information/documents  used  are  described 
below: 


a.  Baseline  Intelligence  Products.  Provides  threat 
information  by  geographic  region/country  on  all  weapons  systems. 
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doctrine,  tactics,  organizations,  equipment,  and  military  forces. 
It  is  continuously  updated  by  DIA  and  approved  by  DIA. 

b.  System  Threat  Assessment  Report  (STAR)  (See  Chapter  13) . 

c.  Integrated  Threat  Tactical  Operations  Plan  (ITTOP) . 

Provides  information  on  all  known  and  projected  threat  force 
mixes.  It  is  used  for  threat  simulators  in  inventory,  provides 
command  and  control;  integrated  force  operations;  and  crew  drill 
and  procedures  (including  ECM/ECCM) .  It  is  updated  as  required 
by  OTSA  with  input  from  S&TI  Centers.  It  is  approved  by  DA 
DCSINT  (crew  drills/procedures  validated  by  the  MATDEV) . 

19-13.  Support  Planning 

a.  The  Challenge.  T&E  is  based  on  Critical  Operational  Issues 
and  Criteria  developed  early  in  the  acquisition  process.  Answers 
to  many  of  these  issues/criteria  are  threat  sensitive  in  that 
they  depend  largely  on  the  threat  environment  to  which  the  system 
will  be  subjected.  Some  of  the  challenges  to  T&E  planners  are; 

(1)  Differences.  Since  development  of  Critical  Operational 
Issues  and  Criteria  precedes  the  development  of  the  detailed 
threat  assessments  in  threat  related  documents,  significant 
differences  can  occur  between  the  documented  threat  and  that  used 
to  develop  Critical  Operational  Issues  and  Criteria. 

(2)  Gaps.  Intelligence  gaps  become  evident  when  a  system 
is  progressively  defined  as  it  proceeds  through  the  acquisition 
process.  These  gaps  generate  both  intelligence  production  and 
collection  requirements,  which,  as  they  are  developed,  may  change 
the  projected  threat. 

(3)  System  Operating  Requirements.  Due  to  the  evolving 
threat,  keeping  the  system  operating  requirements  in  consonance 
with  the  threat  is  sometimes  difficult.  However,  failure  to  do 
so  can  result  in  an  unrealistic  T&E  threat  representation. 

(4)  Inaccuracies.  TEMP  and  OTP  development  precedes  that 
of  the  Threat  TSP  and  can  result  in  inaccuracies  and/or 
inadequacies  in  projections  of  assets  required  for  test  threat 
representation. 

b.  Critical  Intelligence  Parameters  (CIP) .  Together,  the 
MATDEV  and  the  intelligence  community  establish  limits  on  how 
much  the  threat  can  change  without  causing  a  major  redesign  or 
reassessment  of  the  program.  These  limits,  expressed  as  CIPs, 
define  thresholds  for  characteristics  of  actual  or  projected 
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threat  systems  (e.g.,  capabilities,  numbers,  types,  or  mixes  of 
systems) ,  which  if  exceeded,  could  substantially  change  a 
system's  operational  requirement.  Once  defined,  CIPs  are 
submitted  through  intelligence  channels  for  validation  and 
subsequent  collection  and  production.  They  are  included  in  the 
STAR  and  Threat  TSP  and  serve  as  a  T&E  planning  factor. 

'  c.  Use  of  the  Threat  TSP  in  Test  Planning.  The  purpose  of  the 
Threat  TSP  is  to  document  the  threat  environment  appropriate  to 
test  a  developing  system.  When  reviewing  the  Threat  TSP,  the 
evaluator  and  tester  must  determine  whether; 

(1)  Threat  overview  in  the  Threat  TSP  adequately  reflects 
the  threat  assessment  of  the  STAR. 

(2)  Threat  scenarios  have  been  validated  and  accurately 
replicate  the  test  threat  environment  needed  to  address  the 
critical  issues. 

(3)  Weapon/target  matrices  adequately  reflect  the  validated 
threat. 

(4)  The  threat  is  appropriately  configured  for  the 
environmental  conditions  and  means  of  employment  (doctrine, 
tactics,  organization,  and  force  structure)  necessary  to  answer 
the  issue  focus  of  the  TEP. 

(5)  Detailed  test  planning  has  been  conducted  with  full 
cognizance  of  CIPs. 

(6)  Targets  and  threat  simulators  are  available  and 
scheduled  to  replicate  the  threat  scenarios  depicted  in  the 
Threat  TSP.  Consideration  must  also  be  given  to  the  use  of 
surrogates,  (in  the  absence  of  appropriate  targets  or  threat 
simulators)  and  the  accreditation  of  all  threat  representative 
systems  in  advance  of  testing. 

19-14.  Validation  and  Accreditation 

Validation  and  accreditation  are  applicable  to  all  threat 
simulators  and  targets  which  are  used  to  represent  a  specific 
threat  system  (or  portion  of  a  specific  threat  system)  in 
developmental  and/or  operational  tests.  Laboratory  simulators 
will  be  validated  and  accredited  if  they  represent  a  part  or 
function  of  a  specific  threat  system  and  are  used  in  a  test 
supporting  a  milestone  decision. 

a.  Validation. 
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(1)  Definition.  Threat  simulators  and  targets  are 
developed  in  order  to  portray  threat  systems  for  identified  test 
support  requirements.  Accordingly,  threat  simulators  and  targets 
may  duplicate  or  represent  a  limited  number  of  attributes  of  the 
threat  system.  Validation  is  the  process  of  comparing  targets 
and  threat  simulators  to  DIA-approved  threat  data  and  documenting 
the  differences  so  they  can  be  factored  into  test  and  training 
planning,  execution,  and  evaluation.  Validation  provides  the 
analysis  necessary  to  justify  continuation  of  development  or  use, 
or  modification  to  achieve  or  restore  a  sufficiently  realistic 
representation.  Thus,  validation  must  be  based  upon  expert 
knowledge  of  the  threat,  the  simulator  or  target,  and  test 
requirements . 

(2)  Validation  Working  Group  (VWG) .  Validation  occurs  and 
is  documented  by  a  VWG  at  the  following  key  decision  points 
during  the  life  cycle  of  a  threat  simulator  or  target;  Design 
Specification  Review  (DSR) ,  Critical  Design  Review  (CDR) , 

Initial  Operational  Capability  (IOC) ,  upon  modification  which 
affects  fidelity,  and  periodically  following  deployment.  The  VWG 
is  chartered  by  TEMA,  chaired  by  the  TEMA  designee,  and  composed 
of  representatives  from  the  responsible  user,  intelligence,  and 
simulator/target  development  organizations.  Mandatory  membership 
includes  representatives  from  TECOM,  OPTEC,  Army  Materiel  System 
Analysis  Activity  (AMSAA) ,  and  PM  ITTS.  "As  Required"  membership 
includes  representatives  from  TRADOC,  Army  Research  Laboratory 
(ARL) ,  MSIC,  FSTC,  AMC  Research,  Development  and  Engineering 
Centers  (RDEC) ,  and  other  Army/Service  organizations. 

(3)  Requirements.  Validation  requires  technical  and  use 
analysis . 


(a)  Technical  Analysis.  Technical  analysis  compares  the 
technical  characteristics  and  capabilities  of  a  threat  simulator 
or  target  to  current  DIA-approved  intelligence  information  on  the 
specific  threat  system.  A  Technical  Analysis  Report  (TAR)  will 
quantify  the  fidelity  of  the  threat  simulator /target  and  the 
threat,  and  state  the  implications  of  differences.  Technical 
analysis  will  use  data  developed  jointly  by  the  technical 
analysis  organization  (e.g.,  an  S&TI  Center)  and  the  simulator/ 
target  developer  to  satisfy  the  DoD  Tri-Service  Executive 
Committee  (EXCOM)  standard  validation  criteria  for  threat 
simulators.  The  responsible  S&TI  Center  will  issue  the  TAR  to 
the  VWG. 

(b)  Use  Analysis.  "Use  analysis,"  accomplished  by  the 
VWG,  compares  the  capabilities  and  limitations  of  the  threat 
simulator  or  target  (described  in  the  TAR) ,  the  threat,  and  the 


19-20 


DA  Pamphlet  73-1,  Part  One,  16  Oct  92  (DRAFT) 


projected  general  use  of  the  system  to  determine  its  utility. 

"Use  analysis"  is  documented  (along  with  the  technical  analysis) 
in  the  validation  report.  Validation  reports  are  submitted  to 
the  Army  CROSSBOW-S  Committee  representative  for  appropriate 
staffing  and  approval. 

b.  Accreditation. 

(1)  Definition.  Accreditation  is  the  process  used  to 
assess  whether  threat  simulators  and  targets  are  suitable  for 
specific  tests.  It  is  essential  for  the  following  reasons; 

(a)  Identifies  differences  (even  those  accepted  during 
development  and  validation)  between  threat  simulators/  targets 
and  the  corresponding  threat  system  which  can  prevent  adequate 
representation  of  the  threat  in  a  particular  test. 

(b)  Ensures  timeliness  of  intelligence  estimates  support 
the  use  of  a  particular  target/threat  simulator  in  a  specific 
test. 

(c)  Identifies  deterioration  and  failures  which  can 
render  a  threat  simulator  or  target  inadequate  to  represent  the 
threat. 

(2)  Approval.  The  accreditation  of  threat  simulators  and 
targets  is  accomplished  by  the  Threat  Accreditation  Working 
Group. 

(3)  Data  Requirements.  In  accreditation  the  data 
requirements  of  a  particular  test  are  compared  to  both  the  latest 
intelligence  estimates  and  to  the  capabilities  of  the  simulator 
or  target  (as  reflected  in  current  validation  reports) . 
Differences  between  the  simulator  or  target  and  the  intelligence 
data  concerning  the  capabilities  of  the  relevant  threat  system 
are  analyzed  and  assessed  against  the  critical ^ test  criteria. 
These  differences  are  also  assessed  against  Critical  Intelligence 
Parameters  (CIP) ,  (defined  in  AR  381-11)  to  determine  whether  the 
performance  characteristics  and  capabilities  of  the  simulator/ 
target  representing  the  threat  are  within  established  CIPs. 
Differences,  particularly  those  exceeding  the  CIPs,  which  cannot 
be  accommodated  or  offset  in  test  planning,  are  defined  and  used 
to  either  justify  modification  of  the  simulator/target  or  support 
the  decision  to  acquire  alternate  simulators  or  targets. 
Differences  found  to  breach  or  exceed  a  CIP  are  reported  to  the 
TIWG/PM  as  a  basis  to  define  and  assess  the  implications  of  an 
improved  threat  capability  on  the  effectiveness,  survivability, 
and  cost  to  the  U.S.  system  to  be  tested. 
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(4)  Timing.  Accreditation  is  required  prior  to  the  use  of 
a  threat  simulator  or  target  and  must  be  incorporated  into  the 
planning  and  preparation  for  tests.  The  accreditation  process 
complements  the  functions  of  the  TIWG  to  improve  test  planning, 
specifically  defines  test  resource  requirements  for  the  OTP,  and 
provides  guidance  for  further  refinement  of  the  Threat  TSP. 

c.  Threat  Accreditation  Working  Group  (TAWG) .  The  TAWG  will 
assess,  document,  and  report  on  threat  simulator /target 
suitability/adequacy  in  support  of  specific  tests. 

(1)  Establishment.  A  TAWG  will  be  established  in  support 
of  the  TIWG  for  each  test  anticipating  use  of  threat  simulators 
and/or  targets  under  the  provisions  of  the  U.S.  Army  Validation 
and  Accreditation  Plan  for  Threat  Simulators  and  Targets. 

(2)  Membership.  TAWG  membership  will  be  tailored  to  the 
U.S.  system  under  consideration  and  will  be  composed  of 
representatives  from  the  responsible  user,  intelligence, 
developer,  and  operator/maintainer  organizations.  The  chairman 
of  the  TAWG  will  be  in  accordance  with  guidance  published  by  TEMA 
in  the  U.S. Army  Validation  and  Accreditation  Plan  for  Threat 
Simulators  and  Targets. 

(3)  Functions.  Functions  and  responsibilities  of  the  TAWG 
are  outlined  below.  These  functions/responsibilities  also 
constitute  the  accreditation  process. 

(a)  Verification.  Compile  and  analyze  test  data 
requirements,  in  comparison  to  system  threat  data,  to  verify  the 
need  for  identified  threat  simulators/targets  and  to  quantify  any 
additional  threat  simulator/target  or  data  requirements. 

(b)  Representation.  Examine  test  data  requirements, 
threat  data  analysis,  and  simulator/target  validation  data  to 
determine  the  ability  of  the  simulator/target  to  adequately 
represent  the  threat  system  under  specific  test  conditions. 

(c)  Documentation.  Document,  to  the  TIWG,  the 
suitability  of  individual  threat  simulators/targets  for  use.  The 
format  for  the  accreditation  report  and  transmittal  letter  are 
contained  in  the  U.S.  Army  Validation  and  Accreditation  Plan  for 
Threat  Simulators  and  Targets.  This  documentation  may  include 
any  specific  test  limitations  imposed  and  recommendations  to  use 
a  particular  threat  simulator /target  as  is,  only  after 
modification,  or  not  at  all.  Additionally,  when  a  particular 
representation  is  required  and  available  assets  are  deemed 
unsuitable,  alternatives  will  be  identified.  Accreditation 
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reporting  will  be  documented  in  the  TEMP  bibliography. 


Section  V 

Inventory  and  Capability  Accounting  and  Use 


19-15.  General 

One  of  PM  ITTS  chartered  responsibilities  is  to  maintain  a 
capability  inventory  of  ITTS  assets  on  hand  at  DoD  and  contractor 
test  facilities  and  a  data  base  of  planned  acquisitions. 
Currently,  the  primary  repository  for  information  on  the  Army's 
test  capabilities  is  the  Test  Facilities  (TESTFACS)  Register. 
Ancillary  data  serves  to  support  the  Director,  Test  and 
Evaluation,  OUSD(A)  T&E  Assets  Data  Base,  and  the  Army  portion  of 
DoD's  Major  Range  and  Test  Facility  Base  (MRTFB) ,  Central  Test 
and  Evaluation  Investment  Program  (CTEIP) ,  Resource  Enhancement 
Program  (REP) ,  and  Operational  Test  and  Evaluation  Capability 
Improvement  Program  (OT&ECIP) .  The  accounting  hierarchy  is 
depicted  in  Figure  19-3. 


(INSERT  FIGURE  19-3  HERE) 

19-16.  Test  Facilities  (TESTFACS)  Register 

TESTFACS  is  an  automated  program  which  lists  and  describes  the 
Army's  test  capabilities.  The  TESTFACS  data  base  consists  of 
Major  Test  Facilities  (MTF)  and  Major  Instrumentation  and  Test 
Equipment  (MITE)  with  values  of  $75K  or  more,  including  targets 
and  threat  simulators/actuals.  The  TESTFACS  data  base  is  updated 
on  a  periodic  basis  by  using  several  options:  manual  input, 
on-line  access  via  the  Defense  Switching  Network  (DSN) ,  or 
personal  computer  floppy  disc  update.  TESTFACS  is  intended  to 
assist  the  Army  in  managing  an  orderly  growth  of  test  capability 
and  serve  as  a  basis  for  test  planners,  including  TIWG,  as  well 
as  providing  a  base  of  information  for  resource  management. 

Manual  input  requirements  and  procedures  are  addressed  in  Figure 
19-4.  POC  -  Project  Manager  for  Instrumentation,  Targets  and 
Threat  Simulators,  ATTN:  AMCPM-ITTS-I ,  Aberdeen  Proving  Ground, 

MD  21005-5001;  DSN  298-2103,  Commercial  (410)  278-2103. 

(INSERT  FIGURE  19-4  HERE) 

19-17 .  Associated  Programs 

a.  OPTEC  Instrumentation  Data  Base  (OIDB) .  The  OIDB  is  an 
automated  inventory  program  that  includes  all  ITTS  assets  owned 
and  operated  by  OPTEC  test  activities.  It  identifies 
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instrumentation  by  category,  class/subclass,  quantity,  and 
location.  Refer  to  Commander,  U.S.  Army  Operational  Test  and 
Evaluation  Command,  ATTN:  CSTE-OPI,  Park  Center  IV  4501  Ford  Ave, 
Alexandria,  VA  22302-1458;  DSN  289-1388,  Commercial  (703) 
756-1388. 

b.  T&E  Assets  Data  Base.  The  Director,  Test  and 
Evaluation,  (D,T&E)  T&E  Assets  Data  Base  is  an  automated 
inventory  which  includes  assets  with  a  value  of  $1M  or  more.  The 
data  base  is  "on-line"  and  accessible  via  T&E  Community  Network 
(TECNET) .  It  serves  to  support  planning  and  to  quantify  DoD 
capability  in  the  form  of  DoDD  3200. 11-D,  MRTFB  Summary  of 
Capabilities.  Refer  to  Director,  Test  and  Evaluation;  DSN 
227-4819,  Commercial  (202)  697-4819. 

c.  Joint  Threat  Simulator  Handbook  (JTSH) .  The  JTSH  is  a 
classified  listing  and  description  of  over  200  threat  simulators 
available  for  use  in  support  of  testing.  Descriptions  include  a 
side-by-side  presentation  of  threat  and  simulator  parametric 
values,  to  provide  an  indication  of  simulator  fidelity,  and,  for 
the  majority,  photographs  of  the  simulator.  Intended  as  a  "first 
look"  data  source,  the  handbook  reflects  inventory  quantity  and 
location  and  identifies  points  of  contact  for  additional 
information.  The  JTSH  is  available  in  hardcopy  or  microfiche 
format  from  the  Joint  Electronic  Warfare  Center,  San  Antonio,  TX, 
or  Personal  Computer  (PC)  compatible  automated  format  through  the 
CROSSBOW-S  Management  Office.  Refer  to  Joint  Electronic  Warfare 
Center,  San  Antonio,  TX  78243-5000;  DSN  969-4705,  Commercial 
(512)977-4705  or  CROSSBOW-S  Management  Office,  Pentagon;  DSN 
788-2802,  Commercial  (205)  842-2802 

d.  Targets  Information  Manual.  This  manual  provides  the 
organization  and  functions  of  the  various  elements  of  the  Targets 
Management  Office  (TMO)  and  a  descriptive  catalog  of  Army  targets 
available  (or  in  development)  for  support  of  T&E  or  training. 
Refer  to  Project  Manager  for  Instrumentation,  Targets  and  Threat 
Simulators,  ATTN:  AMCPM-ITTS-Q,  Redstone  Arsenal,  AL  35898-5798; 
DSN  746-8678,  Commercial  (205)  876-8678. 


Section  VI 

Schedule/Use  Requirements 


19-18.  General 

This  section  describes  procedures  to  be  followed  when  a  user  has 
a  need  for  existing  instrumentation,  targets  (including  foreign 
materiel) ,  and/or  threat  simulators  which  are  not  organic  to  the 
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parent  test  activity/directorate.  It  also  discusses  the 
procedures  to  follow  when  a  need  exists  for  new  instrumentation, 
targets,  or  threat  simulators. 

19-19.  Instrumentation 

Individual  test  activities,  directorates,  ranges,  and/or 
laboratories  possess  organic  instrumentation  assets  consistent 
with  their  mission  focus  (e.g.,  tank  automotive,  electronics, 
missiles,  combined  arms,  etc.). 

a.  Scheduling.  Scheduling  of  organic  assets  is  effected  in 
consonance  with  internal  operating  procedures.  Scheduling  of 
assets  from  external  sources  is  effected  by  direct  coordination 
between  the  borrower  and  lender. 

b.  Conflict  Resolution.  Conflicts  due  to  competing  customer 
requirements  and  availability  of  assets  will  be  resolved  at  the 
lowest  level  possible.  Unresolved  conflicts  will  be  addressed  by 
the  Test  Schedule  and  Review  Committee  (TSARC) ,  within  the  scope 
of  its  charter,  or  by  TEMA. 

c.  Issue/Turn-In  Procedures.  Issue/Turn-In  of  instrumentation 
assets  is  normally  handled  through  property  book  officer 
channels.  Borrowed  assets  will  be  returned  in  their  original 
condition,  unless  other  arrangements  are  pre-coordinated  with 
the  lender. 

d.  Costs.  Costs  associated  with  instrumentation  use  are 
normally  limited  to  those  of  lease,  round  trip  transportation 
(for  borrowed  instrumentation) ,  and/or  any  modifications  required 
for  unique/special  applications,  interface  requirements,  etc. 

The  latter  are  typically  charged  to  the  customer  (e.g.,  PEO/PM) . 
Costs  will  be  reflected  in  the  Outline  Test  Plan  (OTP)  for  TSARC 
approved  tests. 

19-20.  Targets 

a.  For  TSARC  approved  tests,  requirements  for  targets  in 
support  of  tests  will  be  included  within  the  OTP.  Individual 
test  activities,  directorates,  ranges,  and/or  laboratories 
possess  limited  organic  target  assets.  The  vast  majority  of 
aerial  and  ground  targets  used  in  support  of  Army  T&E  are, 
however,  developed/procured,  maintained  in  inventory,  and 
operationally  supported  by  TMO.  The  procedures  of  this  section 
therefore  focus  on  TMO.  Specific  procedural  requirements  for 
assets  held  by  other  organizations  should  be  coordinated  directly 
with  the  appropriate  POC. 
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b.  Scheduling.  Requests  for  use  of  assets  controlled  by  TMO 
will  be  documented  on  SMI  Form  1209) .  Requests  should  be 
submitted  as  soon  as  requirements  are  identified  to  allow  TMO 
sufficient  time  to  coordinate  the  schedules  for  the  target 
hardware  and  associated  support  services.  However,  firm  targets 
support  requirements  for  the  budget  year  and  revised  estimates 
for  future  years  should  be  submitted  to  TMO  NLT  1  April  annually 
to  ensure  target  support  (flight  services)  contracts  are  in  place 
on  1  October.  A  Flow  Diagram  depicting  the  processing  of  a 
request  is  shown  in  Figure  19-5.  Refer  to  Project  Manager  for 
Instrumentation,  Targets  and  Threat  Simulators,  ATTN: 
AMCPM-ITTS-Q,  Redstone  Arsenal,  AL  35898-5798;  DSN  746-3990, 
Commercial  (205)  876-3990. 

(INSERT  FIGURE  19-5)  HERE 

c.  Conflict  Resolution.  Conflicts  due  to  competing  customer 
requirements  and  availability  of  assets  will  be  resolved  at  the 
lowest  level  possible  (either  TMO  or  PM  ITTS) .  Unresolved 
conflicts  will  be  addressed  by  the  Test  Schedule  and  Review 
Committee  (TSARC) ,  within  the  scope  of  its  charter,  or  by  TEMA. 
TEMA  will  coordinate  with  the  Joint  Commanders  Group  for  T&E 
(JCG[T&E3)  to  resolve  any  joint  test  requirements. 

d.  Issue/Turn-In  Procedures. 

(1)  Aerial  Targets.  Aerial  targets  used  in  T&E  are  mostly 
government  owned,  contractor  operated.  The  TMO  furnishes  trained 
contractor  technicians  who  provide  target  presentations  as  a 
turnkey  service.  At  an  unusual  site,  the  tester  may  need  to  be 
concerned  with  on-base  facilitization  of  target  operations.  For 
the  few  aerial  targets  which  are  troop-operated,  the  tester  must 
make  arrangements  for  support  personnel  and  requisition  the 
targets  through  the  supply  system. 

(2)  Ground  Targets.  Ground  targets  are  provided  from  a 
centralized  pool  of  vehicles  at  various  ranges.  The  ranges  which 
normally  operate  ground  targets  will  have  government  or  contract 
personnel  trained  in  their  operation  and  maintenance.  Other 
sites  must  either  arrange  for  loan  of  personnel  or  obtain  their 
own  support.  Issue  and  turn-in  are  arranged  through  a  standard 
loan  agreement.  In  general,  the  target  user  (tester)  is 
responsible  for  repair  of  undestroyed  targets  prior  to  return. 
These  new  procedures  are  in  the  process  of  being  optimized. 

e.  Costs.  Funding  of  T&E  targets  beyond  development  is 
generally  borne  by  the  customer  (i.e.,  the  using  Program/Project 
Manager  or  tester) .  Certain  exceptions  are  made  for  training 
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targets  which  can  support  production  testing.  A  funding  logic 
flowchart  is  shown  in  Figure  19-6.  Since  many  T&E  targets  are 
for  special  purposes  and  not  normally  held  in  inventory  in 
anticipation  of  needs,  they  must  be  funded  by  the  customer  with 
procurement  lead-time.  This  typically  means  that  customer 
funding  is  needed  about  two  years  prior  to  the  supported  test. 

For  TSARC  approved  tests,  cost  associated  with  targets  will  be 
included  within  the  OTP. 

(INSERT  FIGURE  19-6  HERE) 

19-21.  Foreign  Materiel 

For  TSARC  approved  tests,  requirements  for  foreign  materiel  will 
be  included  within  the  OTP.  Traditionally,  foreign  materiel 
available  for  use  by  the  T&E  community  has  been  acquired  under 
the  auspices  of  AR  381-26,  Army  Foreign  Materiel  Exploitation 
Program,  and  held  by  the  Foreign  Science  and  Technology  Center 
(FSTC) .  International  events  of  the  past  several  years,  however, 
have  resulted  in  the  availability  of  an  unprecedented  number  of 
foreign  military  assets  and  overloaded  the  FSTC  capacity.  These 
assets  (i.e.,  foreign  ground  combat  vehicles),  many  of  which  have 
been  designated  for  use  as  targets  by  the  T&E  community,  are 
currently  in  the  possession  of  AMC,  with  PM  ITTS  designated  as 
the  management  agent.  The  Targets  Management  Office  (TMO)  will 
execute  the  activities  associated  with  the  management  of  foreign 
materiel.  Under  the  reliance  concept,  these  assets  will  be 
available  for  use  to  all  members  of  the  DoD  community.  A  Central 
Asset  Pool  (CAP),  planned  for  Yuma  Proving  Ground  (YPG) ,  Arizona, 
will  serve  as  a  central  storage  facility  and  center  of  expertise 
for  storing,  preparing  for  use,  and  shipping  of  foreign  assets. 
Detailed  procedures  for  management,  control,  operation,  and 
support  of  foreign  materiel  are  currently  under  development; 
however,  for  interim  use,  the  procedures  described  below  apply. 

a.  Scheduling.  TMO  will  provide  central  control  over  foreign 
assets  by  coordinating  asset  use  and  maintaining  accountability. 
Requests,  submitted  via  a  Foreign  Equipment  Request  to  TMO,  will 
be  considered  based  on  asset  availability  and  priority  of  need  as 
indicated  above  for  targets.  All  requests  for  destructive 
testing  will  be  subject  to  approval  by  the  Army  Foreign  Materiel 
Review  Board  (AFMRB) .  All  operation  and  maintenance  (O&M)  of 
foreign  materiel  must  be  performed  by  CSTA-trained  personnel. 
Thus,  provisions  must  be  made  at  the  time  of  scheduling  to  ensure 
that  certified  personnel  will  be  available  to  conduct  required 
O&M  functions.  While  this  requirement  may  impose  some 
restrictions  on  scheduling  flexibility,  these  procedures  are 
necessary  to  maximize  the  safety  of  test  personnel  and  ensure 
adequate  operational  fidelity  while  minimizing  potential  damage 
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to  the  equipment.  Refer  to  Project  Manager  for  Instrumentation, 
Targets  and  Threat  Simulators,  ATTN:  AMCPM-ITTS-Q,  Redstone 
Arsenal,  AL  35898-5798;  DSN  746-3099/  788-6797,  Commercial  (205) 
876-3099/842-6797. 

b.  Conflict  Resolution.  Requests  for  use  of  foreign  materiel 
will  be  prioritized  using  the  criteria  listed  below.  Conflicts 
due  to  competing  customer  requirements  and  availability  of  assets 
will  be  resolved  at  the  lowest  level  possible  (either  TMO  or  PM 
ITTS) .  Unresolved  conflicts  will  be  addressed  to  TEMA  for 
resolution.  TEMA  will  also  coordinate  with  the  Joint  Commanders 
Group  for  T&E  (JCG[T&E])  to  resolve  any  joint  test  requirements. 

(1)  Satisfying  many  versus  few  users. 

(2)  Short  term  versus  long  term  allocation  of  assets. 

(3)  Non-destructive  versus  destructive  testing. 

(4)  DCSOPS  weapon  system  priority. 

c.  Issue/Turn-In  Procedures.  After  approval  of  a  foreign 
military  asset  use  request,  TMO  will  direct  the  CAP  to  prepare 
and  ship  the  foreign  asset (s)  to  the  requested  location.  Unless 
otherwise  directed  by  TMO,  the  user  will  return  the  foreign 
asset (s)  to  the  CAP  after  use.  A  loan  agreement  will  be  used  to 
define  responsibilities  and  conditions  for  the  use  of  foreign 
assets.  A  sample  is  shown  at  Figure  19-7. 

(INSERT  FIGURE  19-7  HERE) 

d.  Costs.  Details  for  funding  the  management,  operations,  and 
support  of  foreign  assets  will  be  developed  as  responsibilities 
of  and  relationships  between  various  support  agencies  and 
customers  are  more  clearly  defined.  A  mixture  of  institutional 
and  customer  funding  will  be  used  for  the  support  of  the  CAP, 
with  the  customer  bearing  all  costs  associated  with  foreign 
materiel  asset  use  (e.g.,  transportation  to  and  from  the  test 
location,  operation  and  support  of  the  asset  on  site,  and 
maintenance  and/or  repair  upon  return  to  the  CAP) . 

19-22.  Threat  Simulators 

For  TSARC  approved  tests,  requirements  for  threat  simulators  will 
be  included  within  the  OTP.  As  defined,  threat  simulators  are  a 
family  of  equipment  used  to  represent  adversary  systems.  Army 
Threat  Simulators  (ATS) ,  developed  by  PM  ITTS  and  subject  to 
provisions  of  the  U.S.  Army  Validation  and  Accreditation  Plan  for 
Threat  Simulators  and  Targets,  are  normally  "fielded"  to  the 
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OPTEC  Threat  Support  Activity  (OTSA)  for  operation  and 
maintenance.  The  procedures  of  this  section  therefore  focus  on 
OTSA.  Specific  procedural  requirements  for  assets  held  by  other 
organizations  (e.g. ,  Electronics  Proving  Ground  [EPG] , 
Vulnerability  Assessment  Laboratory  [VAL] ,  and  TEXCOM 
Intelligence  and  Electronic  Warfare  Test  Directorate  [lEWTD]) 
should  be  coordinated  directly  with  the  appropriate  POC. 

a.  Scheduling.  The  OIDB  provides  a  list  of  threat  simulators 
available  at  OTSA.  Additional  assets  and  information  are 
addressed  in  the  Joint  Threat  Simulator  Handbook.  Scheduling  of 
OTSA  held  threat  simulators  is  accomplished  directly  with  OTSA 
and  should  be  effected  no  later  than  24  months  in  advance  of  the 
required  test  date.  Formal  schedule  coordination  and  approval 
for  use  is  conducted  as  a  part  of  the  TSARC  process.  Priority 
for  use  is  based  on  the  hierarchy  listed  below.  Refer  to 
Director,  U.S.  Army  Operational  Test  and  Evaluation  Command 
OPTEC)  Threat  Support  Activity  (OTSA),  ATTN:  CSTE-OPO,  Fort 
Bliss,  Texas  79916-0058;  DSN  978-3123/7709,  Commercial  (915) 
568-3123. 

(1)  TSARC  approved  test. 

(2)  Non-TSARC  Army  test. 

( 3 )  Non-Army  test . 

(4)  Training  support. 

b.  Conflict  Resolution.  OTSA  will  address  conflicts  on  a  case 
by  case  basis  and  develop  a  threat  simulator  usage  schedule  that 
best  accommodates  all  parties  involved.  Conflicts  that  cannot  be 
resolved  at  the  OTSA-customer  level  will  be  elevated  for 
resolution  to  the  TSARC,  within  the  scope  of  its  charter,  or  to 
TEMA. 


c.  Issue/Turn-In  Procedures.  Threat  simulators  held  in 
inventory  by  OTSA  are  not  "issued”  for  use  by  the  T&E  community. 
Instead,  OTSA  personnel  execute  all  aspects  of  threat  simulator 
activity  during  testing.  This  is  done  because  of  the  requirement 
for  trained  operator  and  maintenance  personnel  to  optimize 
performance  of  the  simulator. 

d.  Costs.  For  all  types  of  test  and  training  support,  OTSA 
will  prepare  a  cost  estimate  and  provide  a  summary  sheet  to  HQ 
OPTEC  (CSTE-OPI)  for  use  in  communication  and  coordination  with 
the  customer.  For  TSARC  approved  tests,  costs  associated  with 
threat  simulator  support  (drawn  from  the  summary  sheet)  will  be 
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included  within  the  Outline  Test  Plan  (OTP) .  For  other  tests  and 
training  support,  the  designated  test  activity  will  ensure  that 
funds  or  Funds  Certification/Availability  letters  are  obtained  to 
underwrite  costs  of  the  required  support.  OTSA  threat  simulator 
support  costing  for  TSARC  approved  tests  includes  TDY  costs 
(including  rental  cars) ;  transportation  of  equipment  (round 
trip);  special  training  of  operators  as  required;  petroleum,  oil, 
and  lubricants;  overtime;  unique  instrumentation,  including 
interfacing,,  data  acquisition,  and  processing  equipment; 
supplies;  repair  parts  associated  with  degradation  of  the 
system;  and  any  additional  personnel  resources  required  to 
support  the  effort.  For  other  test  and  training  support,  the 
customer  will  also  provide  funds  for  core  contract  costs  (those 
associated  with  the  support  contractor's  normal  day-to-day 
operations  including  base  pay  and  overhead)  during  the  required 
support  time  frame  as  determined  by  OTSA,  and  a  usage  fee  based 
on  maintenance  cost  per  day  (X)  number  of  days  (X)  10%. 

Section  VII 

Development  and  Acquisition  Requirements 
19-23.  General 

The  process  discussed  in  this  section  provides  general 
information  for  the  tester  and  instrumenter  who  is  unable  to 
fulfill  needs  from  inventory.  Throughout  the  process, 
requirements  are  discussed  as  though  they  pass  through  each 
review  favorably.  An  actual  requirement  may  fail  to  pass  through 
a  certain  review,  and  may  be  passed  to  an  earlier  point  in  the 
cycle  for  revision  or  reconsideration  and  subsequent  resubmission 
or  deletion.  It  must  be  noted  that  the  ITTS  acquisition  process, 
from  the  identification  of  a  need  to  the  availability  of  a 
fielded  system,  routinely  takes  three  to  five  years  and  may  take 
significantly  longer  (e.g.,  10-12  years)  for  certain  major 
systems.  For  this  reason,  Requirers  must  adjust  their  planning 
and  resourcing  efforts  accordingly  for  immediate  needs. 

19-24.  Relationships 

Various  programs  (e.g.,  sustaining  instrumentation)  are  funded 
and  pursued  at  the  command  level.  Key  among  these  are  the  TECOM 
Technology  Development  and  Acquisition  Program  (TDAP) ,  which 
calls  for  TECOM  test  centers  to  relate  shortfalls  (and  therefore 
projects)  to  specific  facilities  in  TESTFACS;  and  the  OPTEC  User 
Test  Instrumentation  Program  (UTIP) .  Requiring  commands  (e.g., 
TECOM  and  OPTEC)  and  PEO/PM  are  encouraged  to  maintain 
communications  with  PM  ITTS  concerning  requirements  to  enable 
rapid  identification  of  any  cases  of  duplication  or  potential 
savings  which  may  be  gleaned  from  combining  the  development  of 
similar  projects.  Further,  communication  with  PM  ITTS  will 
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facilitate  community-wide  awareness  of  instrumentation,  target, 
and  threat  simulator  developments  and  thereby  enhance 
inter-command  coordination  and  technology  sharing.  The  result 
will  be  reduced  costs  and  a  more  effective  and  efficient  Army 
ITTS  program.  Due  to  significant  changes  ongoing  throughout  the 
Army,  major  changes  to  this  process  will  occur  in  the  near-term 
(i.e.,  next  several  months  to  a  year).  Currently,  the  process, 
as  described  is  an  accurate  representation  of  the  procedure  to 
follow  when  requesting  ITTS  developments.  As  major  changes 
occur,  the  process  description  will  be  updated  to  reflect  those 
changes . 

19-25.  Process 

Each  step  of  the  ITTS  requirements  process  is  identified  by  the 
documents  which  are  input  to  the  process,  a  brief  statement  of 
the  actions  in  the  process,  and  a  description  of  the  documents 
which  are  output  from  the  process.  The  Originators  and 
Recipients  are  listed  where  applicable.  Because  some  steps  of 
the  process  differ  for  the  development  of  instrumentation  versus 
that  of  targets  and  threat  simulators,  this  section  is  divided 
into  three  distinct  subsections:  one,  steps  unique  to  the 
instrumentation  cycle;  two,  steps  unique  to  the  target  and  threat 
simulator  cycle;  and  three,  steps  common  to  both  cycles.  A 
composite  graphic,  depicting  the  full  cycle,  is  presented  at  the 
end  of  the  section. 

19-26.  Instrumentation  Cycle 

a.  Requirements  Document  Generation  (See  figure  19-8) .  The 
Originator  (usually  a  test  directorate  or  test  center; 
occasionally  a  command  headquarters  or  program/project  manager, 
which  may  also  serve  as  the  Requirer)  generates  the  requirement 
document.  Major  instrumentation  requirements  are  documented  in 
accordance  with  DoD  5000. 2-M.  Sustaining  instrumentation 
requirements  are  documented  in  the  format  of  the  parent  command 
(e.g.,  TECOM  or  OPTEC) .  Prior  to  this  step,  the  Originator  has 
verified  that  the  need  is  valid  and  determined  that  a  development 
effort  is  necessary  through  a  documented  reference  to  the  Army 
Technology  Base  Master  Plan,  Five  Year  Test  Program  (FYTP) , 
various  Test  Integration  Working  Group  (TIWG)  minutes,  the  Test 
and  Evaluation  Master  Plan  (TEMP) ,  or  any  other  applicable 
documents.  The  Evaluator  and  PM  ITTS  participate  as  required. 

In  summary,  the  lead  organization  is  the  Originator  (occasionally 
the  Requirer) ;  the  Originator  generates  requirement  and  performs 
preliminary  analysis;  the  inputs  are  verbal  or  written 
statements  of  needs  which  may  impact  the  T&E  community;  the 
outputs  are  the  ORD  (major) ,  or  requirement  document 
(sustaining);  and  the  recipient  is  the  Requirer.  Needs  are 
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derived  primarily  from  the  sources  listed  below. 

(1)  Current  and  planned  testing  of  new  technology, 
materiel,  tactics,  or  doctrine.  These  needs  should  be  identified 
by  TIWGs  and  documented  in  TEMPs  at  the  earliest  possible  stage 
of  development.  Test  technology  needed  to  maintain  state-of-the- 
art  capability  is  usually  identified  in  reports  such  as 
Technology  Base  Master  Plan  (TBMP) ,  Next  Generation/ Future 
Systems  (NG/FS)  Handbook,  AMC  Future  Test  and  Evaluation 
Requirements  (AFTER)  Report,  and  U.S.  Army  Research  Laboratory 
(ARL)  reports . 

(2)  Major  Range  and  Test  Facility  Base  (MRTFB) 
Infrastructure.  Needs  which  support  DoD-directed  testing 
capabilities  are  identified  by  test  activities  and  documented  in 
reports  such  as  Long  Range  Business  and  Modernization  Plans. 

(INSERT  FIGURE  19-8  HERE) 

b.  Requirement  Review  and  Consolidation  (See  figure  19-9) . 

The  Requirer  reviews  all  requirements,  checks  for  unwarranted 
duplication,  and  confirms  adherence  to  the  command  long  range 
plan.  Following  consolidation  of  all  submissions,  the  Requirer 
prioritizes  sustainment  requirements  for  execution,  in  consonance 
with  available  funding,  and  identifies  and  nominates,  in  ORD 
format,  potential  major  systems  for  PM  ITTS  management.  In 
summary,  the  lead  organization  is  the  Requirer;  the  Requirer 
consolidates  Command  requirements;  inputs  are  ORD  (major) ,  or 
requirement  document  (sustaining) ;  the  outputs  are  the 
requirements;  and  the  recipient  is  PM  ITTS. 

(INSERT  FIGURE  19-9  HERE) 

c.  ORD  Review  and  Passage  of  Sustaining  Requirements  to  PM 
ITTS  (See  figure  19-10) .  PM  ITTS  reviews  the  ORD  and  coordinates 
with  the  Requirer  to  gain  an  understanding  of  the  need.  PM  ITTS 
gains  visibility  of  the  sustaining  requirements  from  the 
information  copies  provided  by  the  Requirer.  In  summary,  the 
lead  organization  is  PM  ITTS;  PM  ITTS  reviews  the  ORD  and 
resolves  any  questions  with  the  Requirer,  while  the  Requirer 
provides  information  copies  of  sustaining  requirements  to  PM  ITTS 
for  subsequent  examination  and  potential  consolidation  among 
requirers;  inputs  are  the  ORD,  and  an  information  copy  of  the 
sustaining  requirements;  the  outputs  is  the  ORD;  and  the 
recipient  is  PM  ITTS. 


(INSERT  FIGURE  19-10  HERE) 
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d.  ORD  Consolidation  (See  figure  19-11).  PM  ITTS  reviews  the 
ORD  to  identify  duplication  or  opportunity  for  common  development 
with  other  ORDs  or  instrumentation.  PM  ITTS  may,  in  coordination 
with  the  Requirer,  develop  a  single  ORD  to  consolidate  multiple 
ORDs  with  identical  or  similar  requirements.  ORDs  identified  for 
consolidation  are  staffed  to  the  affected  Requirers  and  the 
entire  list  of  ORDs  are  staffed  to  the  various  Requirers 
throughout  the  Army  for  comparative  review.  PM  ITTS  also  advises 
affected  Requirers  of  any  potential  for  combining  sustaining 
requirements.  In  summary,  the  lead  organization  is  PM  ITTS;  PM 
ITTS  reviews  the  ORD  and  sustaining  requirements  and,  where 
opportunities  for  common  development  exist,  PM  ITTS  combines 
requirements  in  accordance  with  requiring  activities;  inputs  are 
the  ORD,  and  information  copies  of  the  sustaining  requirements; 
the  outputs  is  the  ORD;  and  the  recipient  is  the  Army 
instrumentation  community. 

(INSERT  FIGURE  19-11  HERE) 

19-27.  Target/Threat  Simulator  Cycle 

a.  ORD  Generation  (See  figures  19-12  and  19-13) .  The 
Originator  (usually  a  PM,  PEO,  evaluator,  OPTEC,  Space  Defense 
Command  [SDC],  TRADOC,  or  TECOM)  generates  an  ORD  in  accordance 
with  DoD  5000. 2-M.  In  summary,  the  lead  organization  is  the 
Originator  (occasionally  the  Requirer) ;  the  Originator  generates 
the  ORD;  inputs  are  the  requirements;  the  outputs  is  the  ORD; 
and  the  recipient  is  PM  ITTS. 

(INSERT  FIGURE  19-12  HERE) 

(INSERT  FIGURE  19-13  HERE) 

b.  ORD  Review  (See  figure  19-14).  PM  ITTS  reviews  the  ORDs 
and  resolves  any  questions  with  the  Originator.  PM  ITTS  also 
reviews  the  ORD  to  identify  duplications  with  other  ORDs.  PM 
ITTS  will  identify  those  requirements  which  may  be  combined  and 
work  with  the  individual  originators  and  requirers  to  consolidate 
those  requirements.  In  summary,  the  lead  organization  is  PM 
ITTS;  PM  ITTS  reviews  the  ORD;  the  input  is  the  ORD;  the 
outputs  is  the  ORD;  and  the  recipient  is  the  Army  ITTS  community. 

(INSERT  FIGURE  19-14  HERE) 

19-28.  Common  to  Both  Cycles 

a.  ITTS  Requirements  Conferences  (See  figure  19-15).  PM  ITTS 
hosts  three  annual  conferences  (one  each  for  instrumentation. 
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targets,  and  threat  simulators)  to  review  and  validate  decisions 
to  consolidate  similar  major  ORDs  from  various  Requirers  within 
the  Army.  Requirers  will  attend  and  Originators  and  Evaluators 
may  attend  to  ensure  that  only  necessary  development  and 
acquisition  efforts  are  initiated  and  that  similar  requirements 
are  incorporated  into  a  single  ORD  wherever  possible.  The 
conferences  also  provide  fora  to  discuss  other  ITTS  issues  as 
appropriate.  In  summary,  the  lead  organization  is  PM  ITTS;  each 
Army  ITTS  community  activity  participates  in  an  individual  PM 
ITTS  requirements  conference  (e.g.,  one  conference  each  for  the 
instrumentation  community,  targets  community,  and  threat 
simulator  community)  to  review  and  validate  consolidation 
decisions;  inputs  are  the  ORDs,  in  original,  revised,  or  combined 
status;  the  outputs  are  the  ORDs;  and  the  Recipient  is  the  GOSC. 
Outputs  from  the  conferences  will  follow  one  of  the  two  paths 
discussed  below. 

(1)  Sustaining  instrumentation  requirements  are  returned  to 
the  Requirer  for  internal  execution. 

(2)  ORDs  (for  targets  and  threat  simulators  [TTS]  and  major 
instrumentation)  are  presented  to  the  GOSC  for  approval  of 
further  planning  actions. 

(INSERT  FIGURE  19-15  HERE) 

b.  GOSC  Review  (See  figure  19-16) .  The  GOSC  reviews  the 
results  of  the  instrumentation  and  TTS  ORD  generation  procedures 
and  authorizes  PM  ITTS  to  conduct  appropriate  planning,  analyses, 
and  reviews.  Note  should  be  made  that  this  GOSC  meeting  is  held 
concurrently  with  the  GOSC  meeting  mentioned  later  in  the  cycle. 

A  single  convening  of  the  GOSC  serves  to  review  and.  approve 
programs  planned  for  future  PM  ITTS  management  and  programs 
already  under  PM  ITTS  management.  In  summary,  the  lead 
organization  is  the  GOSC;  the  GOSC  reviews  the  requirements; 
inputs  are  the  ORDs  from  both  TTS  and  instrumentation  cycles; 
the  output  is  approval  for  planning;  and  the  Recipient  is  PM 
ITTS. 


(INSERT  FIGURE  19-16  HERE) 

c.  Preliminary  Analyses  (See  figure  19-17).  PM  ITTS,  in 
concert  with  the  Originators  and  Requirers,  performs,  contracts, 
or  delegates  preliminary  trade-off  analyses  for  each  ORD. 

Examples  of  analyses  include  the  COEA,  Economic  Analysis  (EA) , 
Comparative  Analysis  (CA) ,  Study  of  Intended  Use  (SIU) , 
Technological  Opportunity  Analysis  (TOA) ,  and  Functional 
Requirements  Identification  (FRI) .  Subsequent  to  these  analyses. 
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a  PTA  is  recommended.  The  lead  organization  is  PM  ITTS;  PM  ITTS 
conducts  preliminary  analyses  and  reviews;  the  input  is  planning 
approval  from  the  GOSC;  the  outputs  are  the  Preliminary  Technical 
Approach  (PTA)  reguirements;  and  the  Recipient  is  PM  ITTS. 

(INSERT  FIGURE  19-17  HERE) 

d.  GOSC  Approval  (See  figure  19-18) .  The  GOSC  reviews 
programs,  which  now  have  an  initial  cost  estimate, 
requirements  approval.  Subsequent  to  GOSC  approval, 
requirements  are  passed  to  the  MSTIRC  solutions  cycle,  OTECC,  and 
CROSSBOW-S  Committee  reviews  for  multi-service  coordination.  The 
requirements  also  enter  the  Program  Objective  Memorandum  (POM) 
and  Long  Range  Army  Materiel  Requirements  Plan  (LRAMRP)  process. 
The  lead  organization  is  the  GOSC;  the  GOSC  approves  the 
requirements;  the  inputs  are  the  Standard  Army  Quad  Chart 
requirements  approval  and  the  PTA;  the  output  is  GOSC  approval; 
and  the  Recipient  is  PM  ITTS. 

(INSERT  FIGURE  19-18  HERE) 

e.  Concept  Baseline  Establishment  (See  figure  19-19) .  PM 
ITTS  initiates  an  IPS  and  other  project  management  documentation 
to  create  a  Concept  Baseline  (cost,  schedule,  performance, 
supportability)  for  each  project.  The  results  of  the  GOSC 
decision  will  be  incorporated  into  the  baseline  development, 
lead  organization  is  PM  ITTS;  PM  ITTS  creates  the  Concept 
Baseline;  the  input  is  the  PTA;  the  output  is  the  Concept 
Baseline;  and  the  Recipient  is  the  milestone  decision  authority. 

(INSERT  FIGURE  19-19  HERE) 


The 


f.  Milestone  I  (MS  I),  Concept  Demonstration  Approval  (See 
figure  19-20) .  The  milestone  decision  authority  (usually  PM 
ITTS,  but  may  also  be  AMC,  DA,  or  DoD  for  special  interest 
programs,  e.g.,  the  Mobile  Automated  Instrumentation  Suite 
fMAISl)  reviews  the  programmatic  documentation  (e.g.,  IPS,  COEA, 
etc.)  with  reference  to  the  ORD.  If  satisfactory,  the  Concept 
Baseline  is  approved.  This  process  continues  through  the 
acquisition  management  steps  (Best  Technical  Approach, 
Development  Baseline,  specification.  Production  Baseline, 
production/deployment,  and  sustainment).  In  summary,  the  lead 
organization  is  the  milestone  decision  authority;  the  Concept 
Baseline  is  approved;  the  input  is  the  Concept  Baseline;  the 
output  is  MS  I  approval;  and  the  recipient  is  PM  ITTS. 


(INSERT  FIGURE  19-20  HERE) 
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g.  Joint  Service  Reviews  (See  figure  19-21).  PM  ITTS  leads 
joint  service  reviews.  The  outputs  are  generally  MSTIRC,  OTECC, 
and  CROSSBOW-S  Committee  findings. 

(INSERT  FIGURE  19-21  HERE) 

(1)  MSTIRC  Solutions  Review.  MSTIRC  reviews  DT&E  and  OT&E 
inputs  for  a  formal  working-level  multi-service  review  of  all 
requirements.  As  a  result,  potential  CTEIP  candidates  and 
multi-service  duplications  are  identified.  MSTIRC  submission 
formats  currently  vary  between  Need  Cycle  inputs  (which  do  not 
include  dollar  values  on  projects)  and  Solution  Cycle  inputs 
(which  consist  of  quad  charts  and  extensions  identifying 
schedules  and  budgets) .  Currently,  MSTIRC  requires  the  use  of 
TELRIP  format  for  input,  although  suggestions  have  been  made  that 
other  formats  may  supersede  TELRIP  in  the  near  future,  and  that 
the  Need/Solution  cycle  split  may  be  converted  to  a  single 
format.  To  accommodate  all  of  these  variations,  the  term 
"required  MSTIRC  submission  format"  is  used  to  define  the  input 
documentation.  In  summary,  the  lead  organization  is  the  MSTIRC; 
the  MSTIRC  reviews  DT&E  and  OT&E  inputs;  the  outputs  are  the 
findings  of  the  MSTIRC;  and  the  recipient  is  PM  ITTS. 

(2)  OTECC  Review  (for  OT&E  requirements  only) .  The  OTECC 
reviews  OT&E  quad  charts  in  coordination  with  other  OT&E 
requirements  from  the  other  Services.  As  a  result,  potential  REP 
candidates  and  multi-service  duplications  are  identified.  In 
summary,  the  lead  organization  is  the  OTECC;  the  OTECC  reviews 
OT&E  inputs;  the  outputs  are  the  findings  of  the  OTECC;  and  the 
recipient  is  PM  ITTS. 

(3)  CROSSBOW-S  Committee  Review  (for  threat  simulator 
requirements  only) .  The  CROSSBOW-S  Committee  reviews  Army  Threat 
Simulator  requirements  in  coordination  with  threat  simulator 
requirements  from  the  other  Services,  and  reports  to  the  EXCOM. 
The  EXCOM  and  CROSSBOW-S  Committee  use  these  findings  to 
recommend  the  lead  service  for  joint  developments  and  identifies 
those  programs  considered  to  be  unnecessarily  duplicative.  In 
summary,  the  lead  organization  is  the  CROSSBOW-S  Committee; 

the  CROSSBOW-S  Committee  reviews  threat  simulator  inputs; 
the  outputs  are  the  CROSSBOW-S  Committee  findings;  and  the 
recipient  is  PM  ITTS. 

(4)  Incorporate  MSTIRC,  OTECC,  and  CROSSBOW-S  Committee 
Findings.  PM  ITTS  adjusts  programs  and  coordinates  with  other 
Services  as  applicable  to  accommodate  MSTIRC,  OTECC,  and 
CROSSBOW-S  Committee  findings.  These  adjustments  include 
participation  in  the  formation  of  Joint  Project  Offices  (JPO) , 
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where  applicable.  In  summary,  the  lead  organization  is  PM  ITTS; 
PM  ITTS  incorporates  findings  from  the  MSTIRC,  OTECC,  and 
CROSSBOW-S  Committee  reviews;  the  inputs  are  the  MSTIRC,  OTECC, 
and  CROSSBOW-S  Committee  findings;  and  the  outputs  are  adjusted 
programs . 

h.  Milestone  IV  (MS  IV),  Major  Modification  Approval.  MS  IV 
reviews  are  scheduled  as  required  at  any  point  after  MS  I 
whenever  the  Requirer  determines  that  modification  or  upgrade  to 
a  target,  threat  simulator,  or  major  instrumentation  system  in 
development  or  production  is  necessary  due  to  additional  required 
test  capabilities.  Modifications  to  systems  not  in  production 
compete  with  other  possible  alternatives  during  a  new  Concept 
Exploration  and  Definition  phase.  In  summary,  the  lead 
organization  is  PM  ITTS;  the  output  modification  approval  to 
determine  phase  point  to  start  upgrade  program.  The  entire 
requirements  process  is  depicted  in  figure  19-22. 

(INSERT  FIGURE  19-22  HERE) 
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TEST  FACILITIES  (TESTFACS)  REGISTER 

Contained  herein  are  TESTFACS  manual  input  forms  (locally 
reproducible)  ,  applicable  completion  instructions,  and  necessary  data 
field  codes. 

1.  AMC  Form  2434-R,  Installation/Activity,  used  to  describe  the 
installation,  laboratory,  proving  ground,  activity,  or  center  being 
reported. 

2.  AMC  Form  2434-1-R,  Major  Test  Facility,  used  to  identify  major 
test  facilities  (MTF)  which  belong  to  the  reporting  installation, 
activity,  etc. 

3.  AMC  Form  2434-2-R,  Major  Test  Facility,  used  to  describe  the  MTF 
which  are  identified  as  line  items  on  AMC  Form  2434-1-R. 

4.  AMC  Form  2434-3-R,  Major  Instrumentation/Test  Equipment,  used  to 
describe  instrumentation/test  equipment  costing  $75K  or  more. 

5.  AMC  Form  2435-R,  TESTFACS  Inventory  Input  Worksheet,  used  to  enter 
inventory  into  the  TESTFACS  database. 


FIGURE  19-4.  TEST  FACILITIES  (TESTFACS)  REGISTER 


CUSTOMER  REQUEST 
FOREIGN  MILITARY  (FM)  ASSET  OR  SURROGATE 


YES 


FOREIGN  EQUIPMENT  REQUEST 


TEST  TITLE;  _ _ 

MATERIEL  PROGRAM: 

SYSTEMS  REQUIRED: 

SYSTEM  NO.  REQUIRED 


DESCRIPTION  OF  SYSTEM  USE: 

'  '  -  I  ’  , 


TIMEFRAME;  REQUIRED  BY: 

LOAN  TIMEFRAME; 

LOCATION;  (SHIPPING  ADDRESS) ; 


DESTRUCTIVE  OR  NON-DESTRUCTIVE:  _ 

IF  DESCRUTIVE,  DESCRIBE  EXPECTED  DAMAGE: 


WHAT  SUBSYSTEMS  NEED  TO  BE  OPERATIONAL? 


ARE  FUNDS  AVAILABLE  FOR:  SHIPPING 

MAINTENANCE 
DAMAGE  REPAIR 
OPERATION 


POINT  OF  CONTACT:  (PERSON  RESPONSIBLE  FOR  SYSTEM) 
NAME 

TELEPHONE  - - - - 
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OPERATIONAL  REQUIREMENTS  DOCUMENT  (ORD) 


Operational  Requirements  Documents  for  ITTS  systems  will 
adhere  to  the  format  and  content  requirements  of  DoD 
5000. 2-M.  Exceptions,  clarification,  and/or  additional 
information/data  requirements  are  set  off  in  boxed  paragraphs 
following  the  DoD  5000. 2-M  text.  Necessary  abbreviation  of 
paragraph  content  is  authorized,  provided  key  project 
elements  are  adequately  addressed.  Brief  rationale  will  be 
provided  for  any  paragraph  determined  to  be  not  applicable. 


OPERATIONAL  REQUIREMENTS  DOCUMENT 
FOR 

PROJECT  TITLE 


The  project  title  will  be  descriptive  of  the  instrumentation 
system  or  threat  system  (or  portion  thereof)  required. 


1.  General  Description  of  Operational  Capability.  Describe  the 
overall  mission  area,  the  type  of  system  proposed,  and  the 
anticipated  operational  and  support  concepts  in  sufficient  detail 
for  program  and  logistics  support  planning.  Include  a  brief 
summary  of  the  Mission  Need  Statement.  If  a  Mission  Need  State¬ 
ment  did  not  precede  the  Operational  Requirements  Document, 
explain  the  process  that  investigated  alternatives  for  satisfying 
the  mission  need  and  developing  operational  requirements. 


System  description  will  make  use  of  mission  related  terms  and 
concentrate  on  the  end  result  or  main  function  that  the  item 
will  perform,  rather  than  on  the  nature  of  its  design. 


2.  Threat.  Summarize  the  threat  to  be  countered  and  the 
projected  threat  environment.  This  threat  information  should 
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reference  Defense  Intelligence  Agency  or  Service  Technical 
Intelligence  Center  approved  documents  and  be  validated  by  the 
Service  Intelligence  Director.  For  major  defense  acquisition 
programs  (acquisition  category  I) ,  reference  the  Defense 
Intelligence  Agency  (DIA) -validated  System  Threat  Assessment 
Report.  In  some  non-war fighting  systems,  the  threat  may  be 
listed  as  not  applicable. 


For  instrumentation,  this  section  should  contain  a  brief 
description  of  applicable  aspects  of  the  threat  the  system  is 
designed  to  capture.  This  can  be  extended  to  include  the 
environment  as  a  threat.  For  threat  simulators  and  targets, 
this  section  should  contain  a  brief  description  of  the  threat 
system  to  be  replicated. 


3.  Shortcomings  of  Existing  Systems.  Describe  why  existing 
systems  cannot  meet  current  or  projected  requirements  (do  not 
describe  a  proposed  system) . 


Description  will  include  what  shortfall  in  testing  or 
training  capability  dictates  the  need  for  the  item.  Annotate 
classified  shortfall  information  to  show  the  document (s)  from 
which  the  shortfall  was  derived.  Identify  specific  tests, 
materiel  systems,  or  technologies  for  which  no  testing/ 
training  capability  exists.  Briefly  discuss  the  impact  if 
the  project  is  not  pursued. 


4.  Capabilities  Reguired.  Identify  performance  (operational 
effectiveness  and  suitability)  capabilities  and  characteristics 
required.  State  in  operational  terms  and  prioritize  if  possible. 
Specify  each  performance  parameter  in  terms  of  minimum  acceptable 
value  (threshold)  required  to  satisfy  the  mission  need  and  a 
performance  objective.  The  objective  should  represent  a  measure- 
able,  beneficial  increase  in  capability  or  operations  and  support 
above  the  threshold. 

a.  System  Performance.  Include  system  performance  para¬ 
meters  such  as  range,  accuracy,  payload,  speed,  mission  reli¬ 
ability,  etc.  Describe  mission  scenarios  (wartime  and  peacetime, 
if  different)  in  terms  of  mission  profiles,  employment  tactics, 
and  environmental  conditions  (all  inclusive;  natural  and 
man-made,  e.g. ,  weather,  countermeasures,  ocean  acoustics,  etc.). 
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Description  will  include  the  type  of  data  that  the  instru¬ 
mentation  system  will  collect,  monitor,  analyze,  and/or 
control  and  specify  the  associated  accuracy  tolerances  as 
bands  of  performance.  Description  will  include  the  essential 
characteristics,  capabilities,  or  components  to  be  included 
in  a  target  or  threat  simulator  to  provide  the  required  level 
of  fidelity. 


b.  Logistics  and  Readiness.  Include  measures  for  mission- 
capable  rate,  operational  availability,  frequency  and  duration  of 
preventive  or  scheduled  maintenance  actions,  etc.  Describe  in 
terms  of  mission  requirements  considering  both  wartime  and  peace¬ 
time  logistics  operations.  Identify  combat  support  requirements 
including  battle  damage  repair  capability,  mobility  requirements, 
expected  maintenance  manpower  and  skill  levels,  and  surge  and 
mobilization  objectives  and  capabilities. 

c.  Critical  System  Characteristics.  Address  electronic 
counter-countermeasures  (ECCM)  and  Wartime  Reserve  Modes  (WARM) 
tequirements ;  conventional,  initial  nuclear  weapons  effects,  and 
nuclear,  biological,  and  chemical  contamination  (NBCC)  surviv¬ 
ability;  natural  environmental  factors  (such  as  climate,  terrain, 
and  oceanographic  factors) ;  and  electromagnetic  compatibility  and 
frequency  spectrum  assignment  for  systems  operating  in  the 
electromagnetic  spectrum.  Define  the  expected  mission  capability 
(e.g.,  full,  percent  degraded,  etc.)  in  the  various  environments. 
Include  applicable  safety  parameters  such  as  those  related  to 
system,  nuclear,  explosive,  and  flight  safety.  Identify  communi¬ 
cations,  information,  and  physical  and  operational  security 
needs. 


Also  included  should  be  measures  to  be  taken  to  harden 
against  induced  environmental  factors  such  as  shock  and 
vibration,  normal  contaminants,  directed  energy,  smoke 
aerosols,  obscurants,  military  operations  in  urban  terrain, 
etc. 


5.  Integrated  Logistics  Support  fILS) .  Establish  organiza¬ 
tional,  intermediate,  and  depot  level  support  objectives  for 
initial  and  full  operational  capability. 

a.  Maintenance  Planning.  Identify  maintenance  tasks  to  be 
accomplished  and  time  phasing  for  depot  maintenance,  including 
programmed  depot  maintenance  and  surveillance  inspections  such  as 
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nuclear  hardness  and  structural  integrity.  Describe  the  planning 
approach  for  contract  versus  organic  repair. 

b.  Support  Equipment.  Define  the  standard  support  equipment 
used  by  the  system.  Describe  the  fault  isolation  capabilities 
desired  of  automatic  test  equipment  at  all  levels,  expressed  in 
terms  of  realistic  and  affordable  possibilities  and  confidence 
levels. 


c.  Human  Systems  Integration.  Briefly  describe  the  opera¬ 
tional  and  maintenance  training  concept  (pipeline,  training 
devices,  embedded  training/onboard  training,  interactive 
courseware) .  Identify  manpower,  personnel,  and  training 
constraints.  Establish  objectives  and  thresholds  if  applicable 
for  manpower  (force  structure  and  end  strength) ,  personnel 
(numerical  and  skill  level) ,  training,  and  safety.  Specify 
manpower  and  training  methodologies  to  be  used  (e.g.,  HARDMAN 
Comparability  Methodology) . 

d.  Computer  Resources.  Identify  computer  resource 
constraints  (examples  include  language,  computer,  data  base, 
architecture,  or  interoperability  constraints) .  Address  all 
mission  critical  and  support  computer  resources,  including 
automated  test  equipment.  Describe  the  capabilities  desired  for 
integrated  computer  resources  support.  Identify  any  unique  user 
interface  requirements,  documentation  of  needs,  and  special 
software  certifications. 

e.  other  Logistics  Considerations.  Describe  the 
provisioning  strategy  for  the  system.  Specify  any  unique  _ 
facility  or  shelter  requirements.  Identify  special  packaging, 
handling,  and  transportation  considerations.  Define  unique  data 
requirements  such  as  engineering  data  for  depot  support  and 
technical  orders  for  the  system  and  depot. 

6.  Infrastructure  Support  and  Interoperability.  Discuss 
interfacing  systems  (at  the  system/ subsystem,  platform,  and  force 
levels) ,  specifically  those  related  to  command,  control, 
communications,  and  intelligence  (C3I) ,  transportation  and 
basing,  and  standardization  and  interoperability.  Identify 
companion  Operational  Requirements  Documents  and  other  Services 
that  may  have  similar  requirements.  Assign  a  joint  potential 
designation  (joint,  joint  interest,  independent). 

a .  Command.  Control.  Communications,  and  Intelligence. 
Describe  how  the  system  will  be  integrated  into  command,  control, 
communication,  and  intelligence  architecture  that  is  forecast  to 
exist  at  the  time  the  system  will  be  fielded.  Include  data 
requirements  (data,  voice,  video),  computer  network  support,  and 
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anti- jam  requirements.  Identify  unique  intelligence  information 
requirements,  including  intelligence  interfaces,  communications, 
and  data  base  support  pertaining  to  target  and  mission  planning 
activities,  threat  data,  etc. 


b.  Transportation  and  Basina.  Describe  how  the  system  will 
be  moved  either  to  or  within  the  theater.  Identify  any  lift 
constraints.  Detail  the  basing  and  associated  facilities 
available  for  training  locations  and  main  and  forward  operating 
bases. 


ITTS  will  normally  be  transported  by  truck  or  rail,  with  the 
exception  of  aircraft,  which  will  normally  be  self-deploying. 
Transportability  by  C-141/C-17  aircraft  will  not  be  routinely 
included  as  a  mandatory  transportability  parameter.  Discus¬ 
sion  will  identify  which  test  organizations,  centers,  ranges, 
etc.  will  receive  the  item  and  the  quantity  to  be  received. 


c.  Standardization.  Interoperability,  and  Commonality. 
Describe  considerations  for  joint  use,  NATO  cross-servicing,  etc. 
Identify  procedural  and  technical  interfaces,  and  communications, 
protocols,  and  standards  required  to  be  incorporated  to  ensure 
interoperability  with  other  Service,  joint  Service,  and  Allied 
systems.  Address  energy  standardization  and  efficiency  needs  for 
both  fuels  and  electrical  power  as  applicable. 


Discussion  will  also  include:  compatibility  and  interface 
requirements  for  existing  major  range  and  test  facility 
instrumentation  and  communication  systems;  compatibility 
requirements  with  operational  characteristics  of  the  materiel 
systems  on  which  an  instrumentation  item  will  be  used,  e.g., 
camouflage,  aerodynamics,  radio/electronic  signature,  etc. 


d.  Mapping.  Charting,  and  Geodesy  Support.  Identify 
cartographic  materials,  digital  topographic  data,  and  geodetic 
data  needed  for  system  employment.  Where  possible.  Defense 
Mapping  Agency  standard  military  data  will  be  used. 

e.  Environmental  Support.  Identify  the  standard  and  unique 
weather,  oceanographic,  and  astrogeophysical  support  required. 
Include  data  accuracy  and  forecast  requirements. 
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Discussion  will  also  include  environmental  quality  control  - 
means  of  protecting  the  environment  from  the  operational 
characteristics  of  the  item. 


7.  Force  Structure.  Estimate  the  number  of  systems  or  sub¬ 
systems  needed,  including  spares  and  training  units.  Identify 
the  platforms  and  quantities  of  these  platforms  (including  other 
Services'  or  Government  agencies'  if  appropriate)  that  will 
employ  the  systems  or  subsystems  being  developed  and  procured  to 
satisfy  this  Operational  Requirements  Document. 

8.  Schedule  Considerations.  Define  what  actions,  when  complete, 

will  constitute  attainment  of  Initial  and  Full  Operational  Capa¬ 
bility  (leave  flexible  for  revision  as  the  program  is  progress¬ 
ively  defined  and  trade-off  studies  are  completed) .  Clearly 
specify  the  operational  capability  or  level  of  performance 
necessary  to  declare  Initial  and  Full  Operational  Capability. 
Include  the  number  of  operational  systems,  operational  and 
support  personnel,  facilities,  and  organizational,  intermediate, 
and  depot  support  elements  that  must  be  in  place.  If  availa¬ 
bility  in  a  specific  time  frame  is  important,  specify  an 
objective  for  initial  operational  capability.  Describe  the 
impact  if  this  objective  is  not  achieved  and  identify  a  window  of 
acceptability  if  appropriate. _ 


This  subparagraph  will  be  titled  "Schedule/Cost  Consider¬ 
ations"  for  ITTS  systems.  Included  will  be  subparagraphs 
addressing: 

a.  Risk  Assessment.  Evaluate  the  risk  associated  with 
availability  of  both  intelligence  inforroation/data,  if 
appropriate,  and  technology. 

b.  Schedule.  Include,  as  a  minimum,  outline  baseline 
planning  for:  Threat  Definition  Document  (TDD) ,  if  appro¬ 
priate;  Preliminary  Design  Review  (PDR) ;  Design  Specification 
Review  (DSR) ;  development  start;  Critical  Design  Review 
(CDR) ;  acceptance  testing;  user  testing;  and  Initial 
Operational  Capability  (IOC)  by  fiscal  year  and  quarter. 

c.  Funding.  Identify  both  development /procurement,  by 
type  and  quantity  per  fiscal  year,  and  sustainment  cost 
estimates. 
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The  following  format  requirements,  applicable  to  ITTS 
Operational  Requirements  Documents,  are  in  addition  to  those 
of  DoD  5000. 2-M. 


9.  Point  of  Contact.  Include  the  name,  full  organizational 
mailing  address,  telephone/FAX  (AUTOVON/DSN  and  commercial) ,  and 
E-mail  address  of  the  principal  POC  for  the  ORD. 


Appendices: 

Mission  Need  Statement  (MNS) .  If  applicable,  append  the  MNS  upon 
which  the  ORD  is  based. 

Threat  System  Summary  -  Simulator/Target  Mission  Analysis.  For 
threat  simulators  and  targets,  provide  a  technical  and  opera¬ 
tional  description  of  the  threat  system  to  be  replicated.  Also 
include  the  operational  mode  summary/mission  profile  for  the 
simulator/  target,  keyed  to  projected  use  factors.  Data  should 
form  the  basis  for  and  document  required  simulator /target 
quantities  referenced  in  paragraph  7,  Force  Structure. 

Rationale.  Provide  full  rationale  for  all  required  system 
capabilities/characteristics  described/addressed  in  paragraph  4. 
Use  of  the  term  "self-explanatory”  is  prohibited. 

Coordination.  List  primary  major  commands,  other  Services, 
allied  nations,  and  activities  with  which  coordination  of  this 
ORD  was  effected.  Provide  full  rationale  for  any  nonacceptance 
of  comments  received. 
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Part  Two 

Test  and  Evaluation  Kaster  Plan  (TEMP) 

Format#  Review  and  Approval  Procedures 

Chapter  1.  General  Procedures 

Section  I 
Introduction 

1-1.  General. 

All  acquisition  programs  are  supported  by  an  acquisition  strategy 
(AS)  reflecting  a  comprehensive  and  efficient  Test  and  Evaluation 
(T&E)  program.  To  accomplish  this  task,  each  acquisition 
program/ system  will  have  a  single  Test  and  Evaluation  Master  Plan 
(TEMP) .  All  programs  require  a  TEMP,  except,  level  VI  information 
systems  and  new  drxigs  and  vaccines  that  fall  under  Title  21,  parts 
50,  56  and  312  of  the  Code  of  Federal  Regulations.  (See  AR  73-1, 
paragraph  7 . 4 . b ) . 

1-2.  Capstone. 

When  a  program  consists  of  a  collection  of  individual  systems 
performing  a  common  function,  using  a  common  capability  or 
performing  a  collective  function,  a  Capstone  TEMP  integrating  the 
test  and  evaluation  program  planned  for  the  entire  system  is 
required.  A  Capstone  TEMP  should  not  exceed  30  pages  including 
pages  for  figures,  tables,  matrices,  etc.  Each  individual  system 
TEMP  annexed  to  the  Capstone  TEMP  is  to  follow  the  basic  content 
of  a  TEMP  and  should  not  exceed  30  pages. 

1-3.  Preparation. 

The  TEMP  is  prepared  by  the  program  manager  (understood  to  include 
project  manager  and  product  manager)  in  conjunction  with  principal 
Test  Integration  Working  Group  (TIWG)  members  and  approved  by  the 
appropriate  TEMP  approval  authority,  when  under  time  and  urgency 
constraints,  the  PM  can  prepare  a  strawman  TEMP  to  be  finalized  by 
the  TIWG. 

a.  The  TEMP  is  the  basic  planning  dociament  for  all  life  cycle 
T&E  related  to  a  particular  system  acquisition  and  is  used  by  all 
decision  bodies  in  planning,  reviewing,  and  approving  T&E 
activity.  Drafters  should  therefore  remain  aware  that  the  TEMP  is 
a  planning  mechanism  required  before  proceeding  to  the  next 
acquisition  milestone.  In  addition,  the  approved  TEMP  is  the 
basic  reference  dociament  used  by  the  T&E  community  to  generate 
detailed  test  and  evaluation  plans  and  to  ascertain  schedule  and 
resource  requirements  associated  with  the  T&E  program.  Since  the 
TEMP  charts  the  T&E  course  of  action  during  the  system  acquisition 
process,  all  testing  which  impacts  on  the  progreun  decisions  is 
outlined  in  the  TEMP. 
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b.  The  TEMP  addresses  the  T&E  to  be  accomplished  in  each 
planned  program  phase  with  the  next  phase  addressed  in  the  most 
detail,  when  Developmental  and  Operational  Testing  are  combined, 
the  TEMP  will  separately  address  the  two  different  categories  of 
test.  Part  III,  Developmental  Test  and  Evaluation  Outline,  will 
detail  the  developmental  test  and  evaluation  portion  of  the  DT/OT 
test.  Part  IV,  Operational  Test  &  Evaluation  Outline,  will  detail 
the  operational  test  and  evaluation  portion  of  the  DT/OT  test. 

c.  The  TEMP  is  a  sumary  document  showing  who,  what,  where, 
when,  why,  and  how  the  critical  technical  parameters  and  critical 
operational  issues  will  be  tested/evaluated.  An  approved  TEMP  is 
required  for  a'n  Outline  Test  Plan  (OTP)  to  be  included  in  the  Five 
Year  Test  Program  (FYTP) . 

d.  The  basic  content  of  a  TEMP  should  not  exceed  30  pages, 
including  pages  for  figures,  tables,  matrices,  etc.  In  addition, 
Appendix  A,  Bibliography;  Appendix  B,  Acronyms;  and  Appendix  C, 
Points  of  Contact,  are  excluded  from  the  30-page  limit  as  are  any 
annexes.  Their  size  should  be  kept  to  a  minimum. 

1-4.  Format. 

Army  policy  requires  that  DoD  5000. 2-M  format  be  followed  for  all 
programs  requiring  a  TEMP.  Within  this  format  the  level  of  detail 
is  unique  for  each  program.  Tailoring  of  TEMP  contents  within 
this  format  is  particularly  encouraged  for  programs  not  requiring 
Army  Secretariat  or  OSD  level  approval.  The  level  of  detail 
required  for  any  TEMP  is  directly  related  to  the  approved  T&E 
strategy  and  the  complexity  of  the  test  and  evaluation  effort 
needed  to  verify  attainment  of  technical  performance,  technical 
specifications,  objectives,  safety,  supportability  and  to  support 
the  evaluation/assessment  of  the  operational  effectiveness  and 
operational  suitability  of  the  system,  it  is  not  directly  related 
to  the  size  of  the  program.  The  content  guidance  contained  in  the 
following  chapters  is  intended  to  assist  the  TIWG  organizations 
and  the  TEMP  approval  authority  in  developing  a  TEMP  that  reflects 
an  adequate  and  efficient  test  and  evaluation  program.  These 
content  guidelines  should  not  be  viewed  as  a  rigid  template  for 
all  programs. 

1-5.  COBA  Interface. 

OSD  memorandxam,  21  February  1992,  Subject:  Implementation 
Guidelines  for  relating  Cost  and  Operational  Effectiveness 
Analysis  (COEA)  Measures  of  Effectiveness  (MOEs)  to  Test  and 
Evaluation,  reemphasizes  policy  contained  in  DoDi  5000.2  regarding 
the  need  to  maintain  linkage  between  the  COEA  and  test  and 
evaluation,  particularly  between  the  MOEs  and  the  performance 
parameters  which  define  the  military  utility  of  a  system.  Chapter 
3  contains  guidance  for  TEMP  Parts  I,  III  and  IV  implementing  this 
policy. 
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Section  II 
Non-Majors 

1-6.  Tailoring. 

Tailoring  guidelines  for  TEMPs  not  requiring  Army  Secretariat  or 
OSD  approval  (generally  ACAT  III  or  IV  materiel  and  Class  V 
Information  Mission  Area  (IMA)  programs)  are  addressed  throughout 
this  volume. 

a.  The  general  format  in  DoD  5000. 2-M  must  be  followed, 
however,  tailoring  is  allowed  to  reduce  development  effort  and 
minimize  size. 

b.  Guidance  includes  a  tailored  review  and  approval  process. 

(1)  Paragraph  2-4  describes  a  coordination  process  for 
obtaining  TIWG  concurrence  that  allows  use  of  video  teleconference 
and  mail  or  facsimile  coordination  to  obtain  TIWG  member 
signatures. 

(2)  Paragraph  2-11  describes  a  unique  staffing  and  approval 
process. 

(3)  The  revision  process  described  in  paragraph  2-14. applies 
only  to  TEMPS  that  are  forwarded  for  Army  Secretariat  or  OSD 
approval . 


c.  Guidance  for  tailoring  the  contents  of  materiel  system 
TEMPS  includes: 

(1)  Part  I,  System  Introduction,  paragraph  (c)  Minimum 
Acceptable  Operational  Performance  Requirements.  It  is  sufficient 
to  reference  the  Operational  Requirements  Document  (ORD) . 

(2)  Part  II,  Integrated  Test  Program  Summary.  The  schedule 
format  in  figure  3-9  does  not  have  to  be  rigidly  followed.  A 
program  schedule  can  be  used  as  long  as  test  events  are 
identified.  Funding  information  is  optional.  TIWG  member 
responsibilities  do  not  have  to  be  described  in  detail. 

Referencing  the  charter  is  sufficient. 

(3)  Part  III,  Developmental  Test  and  Evaluation  Outline,  d. 
Live  Fire  Test  and  Evaluation.  Most  ACAT  III  and  IV  programs  will 
not  undergo  formal  live  fire  test  unless  they  meet  the  definition 
of  a  major  covered  program  or  major  munition  as  described  in  Part 
Five,  Live  Fire  Test  and  Evaluation  Guidelines.  For  these  programs 
this  paragraph  is  not  applicable.  This  should  not  be  confused 
with  gun  firing  or  armor  plate  tests,  etc,  needed  to  validate  the 
vulnerability /lethality  requirements  of  the  system. 
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Section  III 
Development 

1-7.  Input 

TEMP  input  is  appropriate  test  and  evaluation  information  that  is 
deemed  necessary  to  ensure  requirements  outlined  from  the  ORD  are 
being  addressed  or  have  been  satisfied.  TEMP  input  is  primarily 
provided  by  the  program  manager,  developmental  tester, 
developmental  evaluator  or  assessor,  operational  tester, 
operational  evaluator,  combat  developer,  and  logistician.  See 
Part  One,  Chapter  8  for  TIWG  composition,  roles  and  functions. 
Other  government  and  contractor  activities  may  also  provide  input 
to  the  TEMP,  when  appropriate.  All  inputs  are  integrated  into  the 
TEMP  by  the  program  manager  who  has  primary  responsibility  for 
preparation,  staffing,  and  update  of  the  temp.  The  TIWG  executes 
a  TEMP  coordination  sheet  that  accompanies  the  TEMP  when  forwarded 
for  TEMP  decision  authority  approval.  A  TEMP  signature  page  is 
executed  by  the  submitter,  reviewers,  and  approval  authority. 

1-8.  OSD  T&E  Oversight. 

The  Office,  Secretary  of  Defense  (OSD)  T&E  Oversight  list  is 
jointly  published  on  an  annual  basis  by  the  Director,  Operational 
Test  and  Evaluation  (DOT&E)  and  the  Director,  Test  and  Evaluation 
{D,T&E),  Office  of  the  Under  Secretary  of  Defense  (Acquisition). 
These  programs  require  OSD  TEMP  approval  and  forwarding  of  T&E 
documentation  to  OSD.  A  preliminary  TEMP  is  due  to  OSD  within  90 
days  after  program  designation.  These  preliminary  TEMPs  will  be 
final  TEMPS  for  programs  in  the  Demonstration-Validation 
acquisition  phase. 

1-9.  Submission. 

Arrry  policy  requires  that  TEMPs  submitted  to  OSD  to  comply  with 
the  milestone  documentation  submission  schedule  and  be  Army 
approved.  DoDi  5000.2  requires  programs  subject  to  Defense 
Acc^isition  Board  (DAB)  review  to  submit  the  TEMP  to  OSD  45  days 
prior  to  the  DAB  committee  review.  Programs  on  the  OSD  T&E 
Oversight  list  that  are  subject  only  to  internal  Army  review  i.e., 
ACAT  1C,  II,  III  and  IV,  must  submit  the  TEMP  to  OSD  45  days  prior 
to  the  Milestone  review.  An  additional  20  days  are  needed  for 
HQDA  and  AMC  staffing  and  DUSA(OR)  approval  prior  to  submission  to 
OSD. 

Section  IV 
TEMP  Update 

1-10.  OSD  T&E  Oversight. 

For  OSD  T&E  Oversight  programs,  when  development  is  complete  and 
critical  operational  issues  are  satisfactorily  resolved,  including 
the  verification  of  deficiency  correction,  a  TEMP  update  is  no 
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longer  required.  A  request  to  delete  the  program  from  the  OSD  T&E 
Oversight  list  should  be  prepared  by  the  PM  and  forwarded  through 
the  PEO  (or  Developing  Agency  if  not  a  PEO  managed  program)  to  The 
U.S.  Army  Test  and  Evaluation  Management  Agency  (TEMA)  for 
forwarding  to  the  Director,  Test  and  Evaluation  (D,T&E)  and 
Director,  Operational  Test  and  Evaluation  (D,OT&E)  for  approval. 
The  request  must  be  coordinated  with  Headquarters,  Training  and 
Doctrine  Command  (TRADOC) ,  Operational  Test  and  Evaluation  Command 
(OPTEC)  and  Army  Materiel  Systems  Analysis  Activity  (AMSAA)  or  the 
Test  and  Evaluation  Command  (TECOM) ,  if  a  TECOM  assessed  program, 
as  the  developmental  independent  evaluator/assessor  and  AMSAA  as 
the  logistician,  before  forwarding  to  TEMA. 

1-11.  Update  deferral 

For  programs  not  on  the  OSD  T&E  Oversight  list,  when  development 
is  complete  and  critical  operational  issues  are  satisfactorily 
resolved,  including  the  verification  of  deficiency  correction,  a 
TEMP  update  is  no  longer  required.  A  request  to  defer  further 
update  should  be  prepared  by  the  PM  or  designated  system  manager, 
coordinated  with  the  TIWG  and  approved  by  the  TEMP  approval 
authority.  Approval  should  be  made  a  matter  of  record.  Programs 
possessing  the  following  attributes  may  no  longer  require  a  TEMP 
update  to  be  submitted: 

a.  Fully  deployed  system  with  no  operationally  significant 
product  improvements  or  bloclc  modification  efforts. 

b.  Full  production  ongoing,  fielding  initiated  with  no 
significant  deficiencies  observed  in  production  qualification  test 
results. 


c.  Partially  fielded  system  in  early  production  phase  having 
successfully  accomplished  all  developmental  test  (DT)  and 
operational  test  (OT)  objectives. 

d.  Programs  for  which  planned  T&E  is  only  a  part  of  routine 
aging  and  surveillance  testing,  service  life  monitoring,  or 
tactics  development. 

e.  Programs  for  which  no  further  OT  or  live  fire  test  (LFT) 
is  required  by  the  Army,  Joint  Chiefs  of  Staff  (JCS),  or  the  OSD. 

f.  Programs  for  which  future  testing  (e.g.,  product 
improvements  or  bloclc  upgrades)  has  been  incorporated  in  a 
separate  TEMP  (e.g.,  an  upgrade  TEMP). 
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Section  V 
Submission 

1-12.  Accompanying  Documents. 

For  all  TEMPS  requiring  OSD  approval,  three  (3)  copies  of  the 
approved  (or  draft  if  not  yet  approved)  Mission  Need  Statement 
(MNS),  Operational  Requirements  Document  (ORD)  and  validated 
System  Threat  Assessment  Report  (STAR)  will  be  forwarded  with  the 
TEMP.  (For  Information  Mission  Area  systems  the  document (s)  to  be 
forwarded  are  the  MNS,  the  functional  description,  when  requested, 
and  the  star  if  prepared  for  the  system) .  If  these  support 
documents  are  final  and  have  not  changed  since  the  last  TEMP 
submission,  a  statement  will  accompany  the  TEMP  attesting  to  that 
fact;  copies  of  the  documents  need  not  be  forwarded.  The 
statement  should  cite  the  date,  version  and  or  change  number  for 
the  most  current  documents. 

1-13.  Referenced  Documents. 

All  documents  referenced  in  the  TEMP  must  be  available  for 
submission  to  Headquarters,  Department  of  the  Arrty  (HQDA)  or  OSD 
on  request. 

1-14.  Preliminary  TEMP. 

For  preliminary  TEMPs,  i.e.,  those  submitted  to  support  Milestone 
I,  information  not  yet  available  should  be  so  noted  and  the  date 
or  event  identified  when  information  will  be  Icnown.  The  TEMP 
should  be  updated  when  the  information  is  available. 
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Chapter  2 

Preparation,  Review  and  Approval  Process 

Section  I 
Introduction . 

2-1.  General 

Development  of  the  TEMP  begins  with  the  establishment  and 
chartering  of  the  Test  Integration  Working  Group  (TIWG)  by  the 
materiel  developer  (for  the  initial  TEMP)  during  the  Concept 
Exploration  and  Definition  phase.  The  TIWG  charter  will  identify 
the  role  and  responsibilities  of  all  agencies  participating  in 
test  and  evaluation.  See  AR  73-1  and  Part  One,  Chapter  8  for 
details,  an  example  of  specific  responsibilities  and  a  sample  of 
the  TIWG  charter.  These  TIWG  specific  responsibilities  are 
aligned  with  the  various  parts  of  the  TEMP  as  shown  in  Figure  2-1. 
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2-2.  Principal  Responsibilities. 

The  program  manager  ultimately  has  the  final  responsibility  to 
produce  the  TEMP.  The  ideal  method  to  develop  a  TEMP  is  for  a 
concurrent  TEMP  development  by  the  program  manager,  the 
developmental  evaluator,  the  developmental  tester,  the  operational 
evaluator,  the  operational  tester  and  the  combat  developer.  On 
small  programs,  or  programs  with  tight  schedules,  it  is  often 
necessary  for  the  program  manager  to  develop  the  first  draft 
strawman  TEMP  with  minimal  or  no  input  from  other  agencies.  That 
input  will  come  during  the  review  cycle  when  the  TEMP  is  staffed 
for  concurrences.  The  responsibilities  to  maintain  TEMP  currency 
and  the  interface  between  TIWG  members  by  TEMP  paragraph  are 
generally  as  shown  in  Figure  2-1. 
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a.  Program  Manager:  prepare  Part  I,  System  Introduction, 
Part  II,  Integrated  Test  Program  Summary,  portions  of  Part  III, 
Developmental  Test  and  Evaluation  Outline  (documenting  tests  that 
provide  information  directly  to  the  sponsor,  e.g.  contractor 
tests)  and  Part  V,  Test  and  Evaluation  Resource  Summary. 

b.  Combat  Developer:  provide  the  Minimum  Acceptable 
Operational  Performance  Requirements,  Part  I,  System  Introduction 
and  input  to  Part  V,  T&E  Resource  Sxammary,  in  particular. 

Manpower/ Personnel  Training  Requirements  and  provides  the  Critical 
Operational  Issues  and  Criteria  (COIC) ,  Part  IV,  Operational  Test 
and  Evaluation  Outline. 

c.  Independent  Developmental  Evaluator /Assessor  and 
Developmental  Tester:  provide  Part  ill.  Developmental  Test  and 
Evaluation  Outline  and  input  to  Part  V,  T&E  Resource  Summary. 

d.  Independent  Operational  Evaluator  and  Operational  Tester: 
provide  Part  IV,  Operational  Test  and  Evaluation  Outline  and  input 
to  Part  V,  T&E  Resources  Summary. 

2-3.  TIWG  Responsibilities. 

The  program  manager  has  overall  responsibility  to  develop  the  TEMP 
to  include  establishing  the  schedule  for  development.  An  early 
TIWG  meeting  should  be  held,  possibly  in  conjunction  with  a  review 
of  the  draft  Operational  Requirements  Document  (ORD)/IMA  systems 
requirements  document,  to  familiarize  TIWG  members  with  the 
preliminary  system  requirements.  This  meeting  is  used  to  assist 
the  program  manager  in  developing  the  test  and  evaluation  strategy 
to  be  incorporated  into  the  acquisition  strategy  and  to  insure 
that  all  appropriate  TIWG  members  are  identified.  The  program 
manager  will  provide  the  available  requirements  documentation, 
draft  acquisition  strategy,  with  the  T&E  strategy  incorporated  and 
other  pertinent  program  documentation  at  this  time.  TIWG  members 
should  be  tasked  to  draft  their  respective  portions  of  the  TEMP. 
The  initial  draft  submission  should  take  no  more  than  30  calendar 
days  for  input  to  the  program  manager. 

a.  The  TEMP  input  received  from  the  TIWG  members  is 
assembled  by  the  program  manager  and  the  consolidated  document  is 
sent  for  review  to  all  TIWG  members  within  15  calendar  days.  An 
additional  30  calendar  day  period  is  allowed  for  the  TIWG  member 
to  staff  the  TEMP  within  the  members  organization  to  ensure 
complete  organizational  concurrence.  Issues  identified  during 
organization  review  and  recommended  changes  should  be  forwarded  to 
the  program  manager  (TIWG  chair)  and  other  TIWG  members  prior  to 
the  TIWG.  Issues  should  be  discussed  and  resolved  at  the  TIWG. 
Electronic  coordination/review  is  encouraged  to  help  meet  the 
tight  review  times.  The  Test  and  Evaluation  Community  Network 
(TECNET)  and/or  local  area  nets  connected  via  the  Defense  Data 
Network  (DDN)  are  available  to  accomplish  the  coordination. 


DA  Pamphlet  73-1,  Part  Two,  16  Oct  92 


(DRAFT) 


b.  TIWG  members  represent  their  organization  and  shall  have 
the  authority  to  concur  in  the  TEMP  for  their  organization.  They 
also  have  an  obligation  to  participate  in  the  TIWG  meeting  unless 
the  agenda  does  not  include  topics  where  they  have  a  direct 
interest. 

c.  It  is  particularly  critical  for  TIWG  members  to  inform 
the  PM  well  in  advance  of  the  TIWG  of  any  issue (s)  which  would 
prevent  concurrence  in  the  TEMP.  There  is  little  value  in 
convening  a  TIWG  for  the  purpose  of  concurring  in  a  TEMP,  if  the 
TIWG  member  cannot  concur  because  of  issue (s)  with  its  content. 

d.  Issues  not  resolved  to  the  satisfaction  of  the  TIWG 
members  are  elevated  through  their  chain  of  command.  If  not 
resolved,  the  issues  are  brought  to  the  attention  of  the  DUSA(OR) 
for  resolution.  This  applies  to  systems  both  materiel  and  IMA  in 
all  ACATs  and  classes. 

e.  The  TEMP  development  and  TIWG  coordination  process  is 
depicted  in  figure  2-2. 


Figure  2-2  TEMP  Preparation /TIWG  Coordination  Process 
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2-4.  TIW6  Meeting  Alternatives. 

It  is  not  necessary  to  conduct  a  TIWG  meeting  only  to  obtain  TEMP 
concurrence  signatures.  The  complexity  and  scope  of  T&E  for  ACAT 
I  and  II  programs  often  warrants  the  travel  time  and  effort 
associated  with  a  TIWG  meeting.  ACAT  III  and  IV  level  and  IMA 
Class  V  programs  may  forgo  the  convening  of  a  TIWG  meeting  and 
conduct  TIWG  business  by  video  teleconference  with  TEMP 
coordination  via  mail.  The  complexity  of  the  T&E  program  should 
dictate  the  TIWG  level  of  effort  and  the  need  for  face  to  face 
discussions.  Means  available  to  facilitate  discussion  and 
coordination  are: 

(1)  Video  teleconference  (VTC) .  Normally  limited  to  1-2 
hours  of  broadcast  time.  Good  for  disseminating  information  and 
reviewing  the  status  of  comments  requiring  changes  to  the  TEMP. 

Not  suitable  for  page  by  page  TEMP  review. 

(2)  Mail  and  facsimile  coordination.  A  viable  way  to  obtain 
TEMP  concurrence  when  the  T&E  program  is  straight  forward  and  non- 
cont rover sial.  A  concerted  effort  is  required  by  all  TIWG  members 
to  forward  concurrences  to  the  PM. 

Section  II 

Review  and  Approval  Process. 

2-5.  General. 

Once  the  TEMP  has  the  concurrence  of  all  the  TIWG  members,  the 
TEMP  is  submitted  for  principal  signatory  review  and  approval. 

This  review  and  approval  process  differs  depending  on  TEMP 
approval  authority.  Changes  required  to  the  TEMP  as  a  result  of 
review  must  be  restaffed  with  the  TIWG  and  other  principal 
signatories.  Restaffing  time  must  be  held  to  a  minimum,  i.e.,  no 
more  than  15  calendar  days.  (See  paragraph  2-5.) 

2-6.  Acquisition  Category  I  (ACAT  I)  and  OSD  T&E 
Oversight  materiel  programs. 

a.  The  program  manager  signs  in  the  "submitted  by"  signature 
block  and  forwards  the  TEMP  to  the  PEO  (developing  agency,  if  not 
under  PEO  structure)  for  concurrence. 

b.  The  PEO  or  developing  agency  forwards  the  TEMP 
rnnrurrentlv  to  HQ  TRADOC  and  OPTEC  for  concurrence.  This 
coordination  process  should  take  no  more  than  30  calendar  days  and 
supplements  the  coordination  accomplished  at  the  TIWG  level. 

c.  The  PEO  forwards  an  original  and  21  copies  of  the  fully 
coordinated  TEMP  to  the  Test  and  Evaluation  Management  Agency 
(TEMA)  for  HQDA  Staffing  and  approval  by  the  DUSA(OR) .  One  copy  of 
the  MNS,  STAR  and  ORD  should  be  forwarded  or  a  statement  of 
currency  if  documents  were  previously  submitted  and  are  still 
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current.  This  coordination  process  should  take  no  more  than  20 
calendar  days. 

d.  Upon  Arrty  approval,  the  PEO  provides  an  additional  15 
copies  to  TEMA  for  forwarding  ty  the  DUSA(OR)  to  the  D,T&E  for 
review  and  OSD  approval.  Also  provide  two  copies  of  the  MNS,  STAR 
and  ORD  or  statement  of  currency  if  documents  were  previously 
submitted  with  the  TEMP  to  OSD  and  are  still  current. 

e.  A  TEMP  is  approved  when  signed  by  the  DOT&E  and  D,T&E. 

The  OSD  objective  is  to  provide  formal  approval  or 

comments /suggested  TEMP  modifications  within  45  calendar  days  of 
receipt. 


f.  The  OSD  approval  memorandum  and  signed  TEMP  signature 
page  are  forwarded  by  TEMA  to  the  PEO  or  developing  agency  for 
inclusion  in  the  TEMP  and  distribution. 

g.  This  process  is  reflected  at  figure  2-3. 

h.  The  signature  page  format  is  shown  in  figure  3-1. 


Figure  2-3.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  I  & 
OSD  Oversight  Materiel  Programs 
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2-7.  Multiservice  ACAT  I  and  OSD  T&B  Oversight  materiel 
programs  for  which  the  Army  has  lead. 

a.  The  program  manager  signs  in  the  "submitted  by"  signature 
block  and  forwards  the  TEMP  to  the  PEO  (developing  agency,  if  not 
under  PEO  structure)  for  concurrence. 

b.  The  PEO  or  developing  agency  forwards  the  TEMP 
rnnnirrpntlv  to  HQ  TRADOC,  OPTEC  and  the  participating  service 
operational  test  agencies  and  participating  service  PEO  or 
Developing  Agency  and  User  Representative  for  concurrence.  This 
coordination  process  should  take  no  more  than  30  calendar  days  and 
supplements  the  coordination  accomplished  at  the  TIWG  level. 

c.  The  PEO  or  developing  agency  provides  an  original  and  21 
copies  plus  one  for  each  participating  service  of  the  TEMP  to  TEMA 
for  HQDA  staffing  and  other  service  approval.  Also  one  copy  of  the 
MNS,  STAR  and  ORD  or  a  statement  of  currency  if  documents  were 
previously  submitted  and  are  still  current.  This  coordination 
process  should  be  accomplished  within  20  calendar  days.  The  T.EMP 
is  then  submitted  for  approval  by  the  DUSA(OR) . 

d.  Upon  approval,  the  PEO  or  developing  agency  provides  15 
copies  of  the  approved  TEMP  to  TEMA  for  forwarding  by  the  DUSA(OR) 
to  the  D,T&E  for  review  and  OSD  approval.  Also  provide  two  copies 
of  the  MNS,  STAR  and  ORD  or  a  statement  of  currency  if  documents 
were  previously  submitted  with  the  TEMP  to  OSD  and  are  still 
current . 

e.  The  TEMP  is  approved  when  signed  by  the  DOT&E  and  D,T&:E. 
The  OSD  objective  is  to  provide  formal  approval  or 

comments /suggested  TEMP  modifications  within  45  calendar  days  of 
receipt.  Each  participating  service  receives  a  copy  of  the  OSD 
memorandum . 

f.  The  OSD  approval  memorandum  and  signed  TEMP  signature 
page  are  forwarded  by  the  DUSA(OR)  to  the  PEO  for  inclusion  in  the 
TEMP  and  distribution. 

g.  This  process  is  reflected  at  figure  2-4. 

h.  The  signature  page  format  is  shown  in  figure  3-2.  If 
there  is  more  than  one  participating  service  or  agency,  a  separate 
Signature  Page  for  each  service/agency  should  be  prepared.  The 
Signature  Page  should  include  the  signature  block  for  the 
Service /Agency  PEO,  the  user  representative,  the  Operational  Test 
Agency  and  the  Service/Agency  TEMP  approval  official.  The  TEMP 
approval  official  for  the  Air  Force  is  the  Assistant  Secretary  of 
the  Air  Force  (Acquisition)  and  for  the  Navy  it  is  the  Assistant 
Secretary  of  the  Navy  (Research,  Development  and  Acquisition) . 
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-PARTCIPATING  SERVICES  RECEIVE  COPY  OF 
OSD  RESPONSE 


Figure  2-4.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  I  & 
OSD  TficE  Oversight  Multiservice  Materiel  Programs,  Army  Lead 

2-8.  Multiservice  ACAT  I  and  OSD  T&E  Oversight  materiel 
programs  for  which  the  Army  Is  a  participant. 

a.  The  TEMP  is  prepared  according  to  Lead  Service  /  Agency 
procedures.  Army  unique  COICs  can  be  provided  for  inclusion  as  an 
annex  to  the  TEMP  when  required  per  DoD  5000. 2-M. 

b.  The  lead  service  program  manager  forwards  the  TIWG  (or 
equivalent)  concurred  in  TEMP  to  the  lead  service  PEO  for 
concurrence.  The  lead  service  PEO  sends  the  TEMP  to  the  Army  PEO 
or  Developing  Agency  for  signature  and  to  secure  OPTEC  and  HQ 
TRADOC  concurrence  on  the  signature  page.  For  those  multi-service 
programs  where  a  separate  Army  TIWG  is  convened  and  TEMP 
coordination  is  documented  on  a  TIWG  Coordination  Sheet,  the 
responsible  Arny  PEO  or  PM  should  forward  the  TIWG  concurrence  to 
TEMA  to  support  HQDA  review  and  approval  by  the  DUSA(OR) . 
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c.  The  lead  service  provides  the  TEMP  to  TEMA  for  HQDA 
staffing  and  approval  by  the  DUSA(OR) .  This  coordination  process 
must  be  accomplished  within  20  calendar  days. 

d.  The  Army  approved  TEMP  is  returned  by  the  DUSA(OR)  to  the 
lead  service. 

e.  The  TEMP  is  forwarded  by  the  lead  service  acquisition 
executive  to  the  D,T&E  for  review  and  OSD  approval. 

f.  The  OSD  approved  TEMP  is  distributed  by  the  lead  service 
PEO.  Each  participating  service  receives  a  copy  of  the  OSD 
memorandum. 


g.  This  process  is  reflected  at  figure  2-5. 


06D  RESPONSE 


Figure  2-5.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  I  & 
OSD  T&E  Oversight  Multiservice  Programs,  Army  Participating 

2-9.  Acquisition  Category  II  (ACAT  II)  and  Army  Special 
Interest  materiel  programs. 

a.  The  program  manager  signs  in  the  "submitted  by"  signature 
block  and  forwards  the  TEMP  to  the  PEO  (developing  agency,  if  not 
under  PEO  structure)  for  concurrence. 
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b.  The  PEO  or  developing  agency  forwards  the  TEMP 
concurrently  to  HQ  TRADOC  and  OPTEC  for  concurrence;  This 
coordination  process  should  take  no  more  than  30  calendar  days  and 
supplements  the  coordination  accomplished  at  the  TIWG  level . 

c.  The  PEO  or  developing  agency  provides  an  original  and  21 
copies  of  the  TEMP  to  the  TEMA  for  HQDA  staffing  and  approval  by 
the  DUS A (OR). 

d.  The  Army  approved  TEMP  is  returned  to  the  PEO  or 
developing  agency  for  distribution. 

e.  This  process  is  reflected  at  figure  2-6. 


f.  The  signature  page  format  is  shown  in  figure  3-3. 


Figure  2-6.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  II 
and  Army  Special  Interest  Materiel  Programs 


2-10.  Nultlservlce  ACAT  II  programs  for  which  the  Army 
has  the  lead. 

a.  The  program  manager  signs  in  the  "submitted  by"  signature 
block  and  forwards  the  TEMP  to  the  PEO  (developing  agency,  if  not 
under  PEO  structure)  for  concurrence. 


b.  The  PEO  or  developing  agency  forwards  the  TEMP 
concurrently  to  HQ  TRADOC,  OPTEC  and  the  participating  service 
operational  test  agencies,  participating  service  PEO  or  Developing 
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Agency  and  User  Representative  for  concurrence.  This 
coordination  process  should  take  no  more  than  30  calendar  days  and 
supplements  the  coordination  accomplished  at  the  TIWG  level. 


c.  The  PEO  or  developing  agency  provides  an  original  and  21 
copies,  plus  one  for  each  participating  service,  of  the  TEMP  to 
TEMA  for  HQDA  Staffing  and  other  service  approval.  The  TEMP  is 
then  submitted  for  approval  by  the  DUSA(OR) . 

d.  The  DUSA(OR)  approved  TEMP  is  returned  to  the  PEO  or 
developing  agency  for  distribution. 

e.  This  process  is  reflected  at  figure  2-7. 

f.  The  signature  page  format  is  shown  at  figure  3-4. 


Figure  2-7.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  II 
Multiservice  Materiel  Programs,  Army  Lead 

2-11.  ACAT  III  and  IV  non -major  Materiel  Programs,  not 
designated  for  OSD  T&E  Oversight  and  Class  V  Information 
Mission  Area  (IMA)  Programs  (to  Include  Multlservlce) . 

a.  TIWG  members  should  staff  the  TEMP  within  their 
organization  to  ensure  complete  review  and  concurrence  during  the 
initial  30  day  TEMP  review  period.  Substantive  issues  should  be 
surfaced  and  resolved  at  the  TIWG.  TIWG  member  concurrence 
constitutes  organization  concurrence. 
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b.  Approval  is  held  in  abeyance  pending  TIWG  member  senior 
management  review.  The  review  period  for  ACAT  III  and  class  V  IMA 
is  20  working  days  and  for  ACAT  IV,  10  working  days  after 
concurrence  by  an  organization's  TIWG  member.  On  expiration  of 
the  review  period,  the  TEMP  approval  authority  signs  the  TEMP  as 
approved  and  executable,  provided  no  objections  are  received  from 
TIWG  organizations.  The  TEMP  approval  authority  is  the  milestone 
decision  authority. 

c.  TIWG  member  organization  can  reverse  their  concurrence 
within  the  designated  review  period  by  providing  written  notice  of 
non- concurrence  signed  by  senior  management.  The  notice  is  to  be 
sent  to  the  program  manager. 

d.  This  process  is  reflected  in  figure  2-8. 

e.  The  signature  page  format  is  shown  in  figure  3-5. 


Figure  2-8.  TEMP  Staffing  and  Approval  Process,  Acquisition  Category  III 
&  IV  Materiel  Programs,  Not  Designated  for  OSD  Oversight  and 
Class  V  Information  Mission  Area  Progreuns  (To  Include 
Multiservice) 

2-12.  Major  Automated  Information  System  Review  Council 
(MAISRC)  programs. 

a.  The  program  manager  signs  in  the  "submitted  by"  signature 
block  and  forwards  the  TEMP  to  the  PEG  or  developing  agency,  if 
not  under  PEG  structure,  for  concurrence. 

b.  The  PEG  or  developing  agency  forwards  the  TEMP  to  GPTEC 
and  the  Proponent /Functional  Agency  or  HQ  TRADGC  for 

Theater /Tactical  systems  for  concurrence.  This  coordination 
process  should  take  no  more  than  30  calendar  days. 
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c.  The  PEO  or  developing  agency  forwards  the  original  and 
all  necessary  copies  of  the  fully  coordinated  TEMP  to  TEMA  for 
HQDA  staffing  and  approval  by  the  DUSA(OR) .  The  number  of  copies 
required  will  be  determined  in  coordination  with  TEMA. 

d.  If  the  system  is  an  OSD  MAISRC  system  the  TEMP  is 
forwarded  by  the  DUSA(OR)  to  the  D,T&E  for  review  and  OSD 
approval . 


e.  This  process  is  reflected  at  figure  2-9  for  Army  MAISRC 
systems  and  at  figure  2-10  for  OSD  MAISRC  systems. 

f.  The  signature  page  format  for  Army  MAISRC  is  at  figure  4- 
1;  the  signature  page  format  for  OSD  MAISRC  is  at  figure  4-2. 


Figure  2-9.  TEMP  Staffing  and  Approval  Process,  Army  MAISRC  Programs 
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Figure  2-10.  TEMP  Staffing  and  Approval  Process,  OSD  MAISRC  Programs 

Section  III  ) 

TEMP  update  and  revision. 

2-13.  Update. 

A  TEMP  update  is  required  to  support  milestone  reviews,  at  program 
baseline  breach,  or  on  occasion  when  the  program  has  changed 
significantly.  The  update  can  be  in  the  form  of  a  complete 
rewrite  of  the  document,  page  changes  or  a  memorandum  indicating 
"no  change".  Page  changes  are  the  preferred  approach  when 
appropriate  because  they  reduce  the  effort  to  review  the  TEMP, 
resulting  in  a  speedier  review  and  approval  process.  Page  changes 
will  be  submitted  as  remove  and  replace  changed  pages  so  as  not  to 
affect  the  integrity  of  the  basic  document.  Coordination  and 
approval  of  the  update  is  done  according  to  the  review  and 
approval  procedures  appropriate  for  the  acquisition  category  and 
TEMP  approval  authority  of  the  program. 

a.  Coordination  and  approval  is  recorded  by  executing  a  TIWG 
Coordination  Sheet  and  a  TEMP  Signature  Page  appropriate  for  the 
program.  Signatures  can  be  obtained  via  facsimile. 

b.  The  initial  submission  date  and  the  current  update  number 
and  date  will  be  shown  on  the  TEMP  cover,  the  TIWG  Coordination 
Sheet  and  Signature  Page. 
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c.  Changes  made  to  an  approved  TEMP  will  be  annotated  by 
change  bars  in  the  outside  margin  of  changed  pages.  A  synopsis  of 
why  specific  changes  were  made  will  accompany  the  update.  When 
page  changes  are  used,  each  changed  page  will  footnote  the  current 
date  and  change  number. 

d.  A  rewritten  TEMP  does  not  require  changes  to  be  noted  by 
change  bars,  but  should  be  accompanied  by  a  synopsis  of  why 
changes  were  made. 

e.  The  "no  change"  memorandum,  when  used  for  ACAT  I  and  II 
and  other  ACATs  designated  for  OSD  T&E  oversight  as  well  as  Army 
and  OSD  MAISRC  programs,  is  prepared  by  the  program  manager,  fully 
coordinated,  and  forwarded  to  TEMA  for  DUSA{OR)  approval  and 
forwarding  to  OSD,  as  appropriate.  Both  the  TIWG  Coordination 
Sheet  and  the  TEMP  Signature  Page  will  be  executed  and  forwarded 
as  enclosures  to  the  "no  change"  memorandum. 

2-14.  Revision. 

A  TEMP  revision  is  required  to  address  comments  received  during 
the  review  and  approval  process  subsequent  to  TIWG  concurrence. 
TEMPS  for  ACAT  III  and  IV  and  IMA  Class  V  programs  are  not  subject 
to  the  procedures  for  revision  unless  they  are  on  the  OSD  T&E 
Oversight  list  and/or  when  senior  management’s  objections  reverse 
the  TIWG  member  concurrence.  A  revision  is  generally  in  the  form 
of  page  changes  although  a  complete  rewrite  of  the  document  may  be 
required  if  the  changes  are  so  substantial  that  page  changes  are 
not  practical.  Page  changes  will  be  submitted  as  remove  and 
replace  changed  pages  so  as  not  to  affect  the  integrity  of  the 
basic  document.  Coordination  and  approval  of  the  revision  is 
according  to  the  approval  procedures  appropriate  for  the 
acquisition  category  and  TEMP  approval  authority  of  the  program. 

a.  For  all  revisions,  TIWG  members  will  be  prpvided  a  copy 
of  the  changes  for  comment  or  concurrence  to  ensure  changes  are 
acceptable.  Verbal  concurrence  will  be  provided  by  all  principal 
TIWG  members  and  recorded  ty  the  TIWG  chairman.  Verbal 
concurrence  will  be  followed  by  a  newly  signed  TIWG  Coordination 
Sheet.  The  intent  of  the  verbal  concurrence  is  to  expedite  TIWG 
level  TEMP  concurrence.  Signatures  can  be  obtained  via  facsimile 
on  separate  pages  for  retention  the  TIWG  chairman. 

b.  A  new  TEMP  signature  Page  will  be  executed  by  the  PM,  PEG 
(or  developing  agency) ,  HQ  TRADOC  or  functional  proponent  for  IMA 
systems  and  OPTEC  for  all  revisions  resulting  from  HQDA  and  OSD 
review. 


c.  The  TEMP  Signature  Page  will  show  the  date  of  the  basic 
document,  the  update  number  and  date,  if  applicable,  and  the 
revision  nvirober  and  date  as  shown  on  the  Signature  Page  format. 
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d.  Changes  made  to  the  TEMP  will  be  annotated  by  change  bars 
in  the  outside  margin  of  changed  pages.  A  brief  synopsis  of  how 
issues  and  comments  were  addressed  and/or  why  specific  changes 
were  made  will  accompany  the  revision.  Each  changed  page  will 
footnote  the  revision  number  and  current  date. 

e.  A  completely  rewritten  TEMP  does  not  require  changes  to 
be  noted  by  change  bars  but  should  be  accompanied  by  a  brief 
synopsis  of  how  issues  and  comments  were  addressed  and/or  why 
specific  changes  were  made  to  the  TEMP. 

f .  The  revision  will  be  forwarded  by  memorandum  to  TEMA  for 
HQDA  review  and  DUSA(OR)  approval  and  forwarding  to  OSD,  as 
necessary.  The  memorandum  will  record  that  TIWG  member 
concurrence  was  obtained  and  will  enclose  the  properly  executed 
TEMP  signature  Page. 

Section  IV 
Administration 

2-15.  Requesting  delay  in  TEMP  submittal. 

The  request  for  delay  for  ACAT  I  and  II,  MAISRC  programs  and  all 
ACATs  designated  for  OSD  T&E  oversight  is  prepared  by  the  program 
manager  and  foirwarded  for  approval  to  the  TEMP  approval  authority. 
The  reason  for  the  delay  must  be  clearly  explained.  Delays  for 
administrative  reasons  are  generally  not  accepted.  The  request 
for  delay  will  be  forwarded  to  TEMA  for  forwarding  to  OSD  or 
DUSA(OR)  approval,  as  necessary. 

2-16.  Publication  considerations. 

TEMPS,  if  bound,  must  allow  for  easy  insertion  of  page  changes; 
spiral  binding,  square  or  glue  bindings  are  not  acceptable.  The 
program  manager  is  responsible  to  provide  the  nxamber  of  copies 
needed  for  HQDA  and  OSD  staff  review.  Quantity  needed  is  as 
previously  identified  in  appropriate  paragraphs  in  Section  II. 
TEMPS  submitted  for  HQDA  and  OSD  approval  must  contain  all 
classified  data  and  appendices/annexes. 
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Chapter  3 

Format  and  Contents  for  Materiel  Programs 

Section  I 
Introduction. 

3-1.  General. 

a.  The  format  for  all  Arny  developed  Test  and  Evaluation 
Master  Plans  will  be  in  accordance  with  Part  7,  DoD  5000. 2-M. 

b.  DoD  5000. 2-M,  Part  7,  attachment  1,  is  replicated  in 
whole  in  the  following  sections,  except  as  noted  below.  Specific 
content  guidance  appropriate  for  Army  TEMP  preparation  is  noted  in 
the  boxed  remarks  following  a  given  paragraph.  Guidance  for  ACAT 
II,  III,  and  IV  programs  is  the  same  as  for  ACAT  I,  except  as 
noted.  Figures  3-8  and  3-9  are  revised  showing  Arny 
interpretation  of  the  DoD  guidance. 

c.  Signature  Page  formats  and  layout  for  programs  by  ACAT 
are  provided  at  figures  3-1  to  3-5.  Program  element  information 
can  be  obtained  from  the  current  year  version  of  AR  37-100-XX. 

d.  An  example  of  a  TIWG  Coordination  Sheet  is  at  figure  3-6. 
The  TIWG  Coordination  Sheet  should  show  the  specific  participants 
of  a  program,  for  example,  the  TIWG  Chair  should  show  the  PM, 
program  name,  the  specific  school/center  should  be  identified  as 
the  combat  developer;  AMSAA  should  be  identified  as  the 
Developmental  Evaluator  or  TECOM  as  the  Developmental  Assessor, 
etc. 

e.  A  TEMP  will  include  a  Signature  Page,  a  TIWG  Coordination 
Sheet  as  shown  in  figure  3-6,  and  an  Outline  as  shown  in  figure  3- 
7. 
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Program  Elements 
Xxxxx 
Xxxxx 


TEST  AND  EVALUATION  MASTER  PLAN  . 

FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (As  applicable) 
REVISION  XX,  DATE  (As  applicable) 


Program  Manager 


DATE 


Program  Executive  Officer 
(or  Developing  Agency, 
if  no  PEO) 


CDR,  U.S.  Arrry  Operational 
Test  &  Evaluation  Command 
(OPTEC) 


DATE 


DATE 


DCS  Combat, 
Doctrine  & 
Development , 
USA  TRADOC 


DATE 


Deputy  Under  Secretary  of 
the  Amy  (Operations  Research) 


DATE 


Director,  Operational 
Test  and  Evaluation 


DATE  Director,  Test  and  Evaluation  DATE 
Under  Secretary  of  Defense 
(Acquisition) 


Figure  3-1.  Signature  Page  format  for  ACAT  I  and  other  ACATs  designated  for 
OSD  test  and  evaluation  oversight. 
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Program  Elements 
Xxxxx 
Xxxxx 


TEST  AND  EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (AS  applicable) 
REVISION  XX,  DATE  (As  applicable) 


Program  Manager 

DATE 

CONCURRE 

:nce 

! 

Program  Executive  Officer 
(or  Developing  Agency  if 

DATE 

Participating  Service  DATE 
PEO  or  Developing  Agency 

no  PEO) 

CDR  U.S.  Army  Operational 
Test  &  Evaluation  Command 

DATE 

Participating  Service 
Operational  Test  Agency 

DATE 

(OPTEC) 

DCS,  Combat,  Doctrine  & 
Developments,  USATRADOC 

COM 

DATE 

PQNENT . A 

Participating  Service 
User  Representative 

PPROVAL 

DATE 

Deputy  Under  Secretary  of  DATE 

the  Amy  (Operations  Research) 

Other  Svc  Acq  Exec 

DATE 

Director,  Operational  DATE  Director,  Test  and  Evaluation  DATE 
Test  and  Evaluation  Under  Secretary  of  Defense 

(Acquisition) 


Figure  3-2.  Signature  Page  format  for  Multiservice  ACAT  I  and  other  ACATs 
designated  for  OSD  T&E  oversight  for  which  Army  is  the  lead 
service . 
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Program  Elements 
Xxxxx 
Xxxxx 


TEST  AND  EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (As  applicable) 
REVISION  XX,  DATE  (As  applicable) 


Program  Manager 


DATE 


Program  Executive  Officer  DATE 
(or  Developing  Agency  if  no  PEO) 


CDR,  U.S.  Army  Operational 
Test  &  Evaluation  Command 
(OPTEC) 


DATE  DCS,  Combat, 

Doctrine  & 
Developments , 
USATRADOC 


DATE 


Deputy  Under  Secretary  of 
the  Army  (Operations  Research) 


DATE 


Figure  3-3.  Signature  Page  format  for  ACAT  II  and  Army  Special  Interest 

programs 
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Program  Elements 
Xxxxx 
Xxxxx 


TEST  AND  EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (AS  applicable) 
REVISION  XX,  DATE  (As  applicable) 


Program  Manager 


DATE 


Program  Executive  Officer 

(or  Developing  Agency  if  no 
PEO) 

DATE 

Participating  Service  DATE 

PEO  or  Developing  Agency 

CDR  U.S.  Army  Operational 
Test  &  Evaluation  Command 
(OPTEC) 

DATE 

Participating  Service  DATE 
Operational  Test  Agency 

DCS  Combat,  Doctrine 
&  Development ,  USATRADOC 

4 

DATE 

APPROVE 

Participating  Service 
User  Representative 

,P  BY 

DATE 

Deputy  Under  Secretary  of  DATE 

the  Arny  (Operations  Research) 

Other  Svc  Acq  Exec 

DATE 

Figure  3-4.  Signature  Page  format  for  Multiservice  ACAT  II  programs  for  which 
Army  is  lead  service. 
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TEST  AND 


EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (As  applicable) 
REVISION  XX,  DATE  (As  applicable) 


SUBMITTED  BY 


Program  Manager  DATE 


APPROVED  BY 


Milestone  Decision  DATE 

Authority 


Figure  3-5.  Signature  Page  format  for  Acquisition  Category  III  &  IV 
programs  not  subject  to  OSD  T&E  oversight  and  Class  V 
Information  Mission  Area  (IMA)  progreuns  (to  include  multiservice 
programs) . 
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TIWG  COORDINATION  SHEET 

TEMP  FOR 

PROGRAM  TITLE 

DATE 

UPDATE  XX,  DATE  (As  applicable) 
REVISION  XX,  DATE  (As  applicable) 

signature 

Date 

Program  Manager 

CONCUR/NONCONCUR 

(TIWG  Chair) 

Combat  Developer/ 

CONCUR /NONCONCUR 

(Proponent  School/Center) 

Developmental  Tester 

CONCUR /NONCONCUR 

(TECOM) 

Develop  Evaluator /Assessor 

CONCUR/NONCONCUR 

(AMSAA/ TECOM) 

Operational  Tester 

CONCUR /NONCONCUR 

(TEXCOM) 

Operational  Evaluator 

CONCUR/NONCONCUR 

(OEC) 

Loaistician 

CONCUR /NONCONCUR 

(AMSAA) 

Sur vi vab i 1 i tv / L e t ha 1 i tv 

CONCUR /NONCONCUR 

(SLAD) 

Threat  Integrator* 

CONCUR /NONCONCUR 

Other  ** 

CONCUR/NONCONCUR 

*  If  Applicable 

**Include  participating  service  representatives  for  multiservice 
programs . 

Figure  3-6.  Sample  TEMP/TIWG  Coordination  Sheet. 
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TEST  AND  EVALUATION  MASTER  PLAN  OUTLINE  (FORMAT) 

Page 

Number 

PART  I  SYST^  INTRODUCTION  (2  pages  suggested  -  refer  to  annexes) 

a.  Mission  Description . . 

b.  System  Threat  Assessment . *  xx 

c.  Minimum  Acceptable  Operational  Performance 

Requirements . . 

d.  System  Description . *.*.*.*.*.**.**.*.*.**.  xx 

Critical  Technical  Parameters  (See  Figure  3-8) xx 

PART  11  INTEGRATED  TEST  PROGRAM  SUMMARY  (2  pages  suggested) 

a.  Integrated  Test  Program  Schedule  (See  Figure  3-9) .  . .  .xx 

b .  Management . . 

PART  III  DEVELOPMENTAL  TEST  AND  EVALUATION  OUTLINE  (10  pages 
suggested) 

a.  Developmental  Test  and  Evaluation  Overview . 

b.  Developmental  Test  and  Evaluation  to  Date . .* ! ! 

c.  Future  Developmental  Test  and  Evaluation . i! 

d.  Live  Fire  Test  and  Evaluation . 


PART  IV  OPEFIATIONAL  TEST  AND  EVALUATION  OUTLINE  (10  pages 
suggested) 


a.  Operational  Test  and  Evaluation  Overview. 

b.  Critical  Operational  Issues . 

c.  Operational  Test  and  Evaluation  to  Date.. 

d.  Future  Operational  Test  and  Evaluation... 


PART  V  TEST  AND  EVALUATION  RESOURCE  SUMMARY  (6  pages  suggested 

a.  Test  Articles . 

b.  Test  Sites  and  Instrumentation . 

c.  Test  Support  Equipment . 

d.  Threat  Systems/Simulators . 

e.  Test  Targets  and  Expendables . !.*!!!!!!.'.*.*! 

f .  Operational  Force  Test  Support . 

g.  Simulations,  Models  and  Testbeds . 

h.  Special  Requirements . !!!!!!!!!!! 

i .  T&E  Funding  Requirements . . ! ! ! ! ! 

j .  Manpower/Personnel  Training . 

APPENDIX  A  Bibliography . . .  a-i 

APPENDIX  B  Acronyms . !!!!!!!!!!!**  B-1 

APPENDIX  c  Points  of  Contact .  . p_i 

ANNEXES /ATTACHMENTS  (if  appropriate)  . 


Figure  3-7.  Test  and  Evaluation  Master  Plan  Outline. 
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Section  II 

TEMP  Format  and  Content  for  Materiel  Systems 

3-2.  TEST  AND  EVALUATION  MASTER  PLAN  CONTENT  (FORMAT) 

PART  I  —  SYSTEM  INTRODUCTION 

a.  Mission  Description.  Reference  the  Mission  Need 
Statement  and  briefly  sximmarize  the  mission  need  described 
therein. 


Define  the  need  in  terms  of  mission,  objectives  and  general 
capabilities. 

Sxammarize  from  Paragraph  2,  Mission  Need  Statement  (MNS) . 

Describe  the  natural  environment  in  two  aspects;  logistically 
and  operationally.  Summarize  from  paragraph  4,  MNS. 


b.  System  Threat  Assessment.  Reference  the  system  threat 
assessment  and  briefly  summarize  the  threat  environment  described 
herein. 


Sxammarize  the  operational  threat  environment  from  paragraph 
4c,  System  Threat  Assessment  Report  (STAR)  and  the  system 
specific  threat  from  paragraph  4e,  STAR.  Include  the  threat  at 
IOC,  follow-on  -  at  IOC  plus  10  and  the  reactive  threat  from 
paragraph  4e  and  4f  (if  applicable) . 

For  ACAT  III  and  IV  programs,  summarize  the  above  information 
from  the  System  Threat  Assessment  (STA) . 


c.  Minimum  Acceptable  Operational  Performance  Requirements. 

Reference  the  Operational  Requirements  Docximent  and  sxammarize 
the  critical  operational  effectiveness  and  suitability  parameters 
and  constraints  (manpower,  personnel,  training,  software,  computer 
resources,  transportation  (lift),  etc.)  described  therein. 


3-9 


DA  Pamphlet  73-1,  Part  Two,  16  Oct  92 


(DRAFT) 


Sxammarize  from  the  Operational  Requirements  Document  (ORD) 
Paragraphs  4,  5,  and  6. 

Discuss  the  relationship  between  the  critical  operational 
effectiveness  and  suitability  parameters  and  the  Measures  of 
Effectiveness  (MOE)  in  the  COEA. 

Operational  requirements  are  specified  in  the  Software 
Requirements  Specification. 

For  ACAT  III  and  IV  programs,  not  designated  for  OSD  T&E 
oversight,  it  is  sufficient  to  reference  the  ORD. 


d.  System  Description.  Briefly  describe  the  system  design. 

Include  the  following  items: 

(1)  Key  features  and  subsystems,  both  hardware  and  software 
(such  as  architecture,  interfaces,  security  levels,  reserves, 
etc.),  allowing  the  system  to  perform  its  required  operational 
mission. 


(2)  Interfaces  with  existing  or  planned  systems  that  are 
required  for  mission  accomplishment.  Address  relative  maturity 
and  integration  and  modification  requirements  for  nondevelopment 
items.  Include  interoperability  with  existing  and/or  planned 
systems  of  other  DoD  Components  or  allies. 

(3)  Critical  system  characteristics  (see  Section  4-C  of  DoD 
Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,")  or  unique  support  concepts  resulting  in  special  test 
and  analysis  requirements  (e.g!,  post  deployment  software  support, 
hardness  against  nuclear  effects;  resistance  to  countermeasures; 
development  of  new  threat  simulation,  simulators,  or  targets.) 


For  MS  I  summarize  from  ORD  or  from  development  specification 
if  available. 

For  MS  II  and  beyond  summarize  from  development 
specification. 

Include  line  drawing  of  system  if  available. 

For  software,  describe  the  overall  system  with  emphasis  on 
where  Mission  Critical  Computer  Resources  (MCCR)  are  used. 

Include  a  single  paragraph  synopsis  of  any  unique  training 
concepts,  logistical  support  concepts,  e.g.,  life  cycle 
contractor  support  and  maintenance  concepts  to  include  planned 
levels  for  maintenance  support. 
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Include  a  description  of  what  constitutes  the  Initial 
Operational  Capability  (IOC)  and  the  Full  Operational  Capability 
(FOC)  for  the  system. 


e.  Critical  Technical  Parameters. 

(1)  List  in  a  matrix  format  (see  Figure  3-8)  the  critical 
technical  parameters  of  the  system  (including  software  maturity 
and  performance  measures)  that  have  been  evaluated  or  will  be 
evaluated  during  the  remaining  phases  of  developmental  testing. 
Critical  technical  parameters  are  derived  from  the  Operational 
Requirements  Document,  critical  system  characteristics  (see  Part  4 
of  DoD  Instruction  5000.2,  "Defense  Acquisition  Management 
Policies  and  Procedures")  and  technical  performance  measures  (see 
Section  6-A  of  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures")  and  should  include  the 
parameters  in  the  acquisition  program  baseline  (see  Part  14,  DoD 
5000. 2-M).  Discuss  the  relationship  between  the  critical 
technical  parameters  and  the  minimum  acceptable  operational 
performance  requirements  in  the  Operational  Requirements  Document. 

(2)  Next  to  each  technical  parameter,  list  the  accompanying 
objectives  and  thresholds  as  illustrated  by  Figure  3-8. 


Critical 

Technical 

Parameters 

Total 

events 

Technical 
objective  for  each 
test  event 

Location 

Schedule 

Decision 

supported 

Demonstrated 

Value 

Measureable 

Parameter 

with 

reference 

Single 
event 
or  test 
phase 

Measureable 
Technical  value 

Test 

facility 

Test 

period 

Milestone 
In-process  review 
or  major  event 

(Include  the 

Actual  Value) 

Detection  range 
10.0  Km 
(reference) 

EOT 

PPT 

PPQT 

7.0  Km 

0.0  KM 
10.0  Km 

ABCRange 
DEP  Range 
DEFRflnge 

.1QFY-XX 

2QFY-XX 

3QFY-XX 

M^ll 

M/Slll 

M/Stll 

X 

Y 

Z 

Figure  3-8.  Sample  Critical  Technical  Parameters  Matrix 

(This  matrix  depicts  the  evaluation  criteria  to  assess 
developanental  progress) 


Critical  Technical  Parameters  -  obtained  from  the  ORD  and 
related  documents  and  discussed  in  the  Acquisition  Program 
Baseline  (APB) . 
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Reference  -  the  source  from  which  the  parameter  and  value  is 
derived. 

Total  events  -  the  developmental  tests  conducted  wherein  the 
parameters  are  tested.  Tests  should  be  outlined  in  Part  III. 

Technical  objective  for  each  event  -  the  value  expected  to  be 
attained  at  that  stage  of  development. 

Location  -  the  place  where  the  test  will  be  performed. 
Normally  a  TECOM  test  facility. 

Schedule'-  the  fiscal  quarter  when  the  test  will  be 
initiated. 

Decision  Supported  -  the  program  milestone  or  review  that 
will  consider  the  results  of  this  test. 

Demonstrated  Value  -  state  the  actual  value  obtained  from 
testing. 

A  MS  I  (preliminary)  TEMP  is  not  expected  to  contain  detailed 
requirements.  The  TEMP  update  to  support  Milestone  II 
(subsequent  to  ORD  approval)  should  include  detailed  values. 


(3)  Highlight  critical  technical  parameters  that  must  be 
demonstrated  before  entering  the  next  acquisition  or  operational 
test  phase  and  ensure  that  the  actual  values  which  have  been 
demonstrated  to  date  are  included  in  the  last  column. 


Critical  technical  parameters  are  defined  as  those  measurable 
critical  system  characteristics  including  software,  that  when 
achieved,  allow  the  attainment  of  the  minimum  acceptable 
operational  performance  requirements. 

Software  critical  technical  parameters  may  include  language, 
architecture,  interfaces,  supportability,  security  levels,  time, 
memory,  and  input /output  reserves.  For  systems  conforming  to 
DoD  STD  2167A,  a  matrix  relating  to  the  critical  technical 
parameters  may  be  found  in  the  Software  Specification 

Discuss  the  relationships  between  the  critical  technical 
parameters  for  test,  the  Measures  of  Performance  (MOP)  in  the 
COEA,  and  the  critical  system  characteristics,  objectives  and 
thresholds  in  the  ORD. 


PART  II  --  INTEGRATED  TEST  PROGRAM  SUMMARY 
a.  Integrated  Test  Program  Schedule 
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(1)  As  illustrated  in  Figure  3-9,  display  on  a  chart 
the  integrated  time  sequencing  of  the  critical  test  and  evaluation 
phases  and  events,  related  activities,  and  planned  cumulative 
funding  expenditures  by  appropriation. 

(2)  Include  event  dates  such  as  milestone  decision 
points;  operational  assessments,  test  article  availability; 
software  version  releases;  appropriate  live  fire  test  and 
evaluation,  and  operational  test  and  evaluation;  low  rate  initial 
production  deliveries;  Full  Rate  Production  deliveries;  Initial 
Operational  Capability;  Full  Operational  Capability;  and 
statutorily  required  reports. 


(3)  A  single  schedule  should  be  provided  for  multi- 
Service  or  Joint  and  Capstone  Test  and  Evaluation  Master  Plans 
showing  all  DoD  Component  system  event  dates. 


Figure  3-9 »  Integrated  Test  Program  .Schedule  (Illustrative  Exeunple) 
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The  integrated  test  program  schedule  will  be  divided  into 
seven  major  areas:  Program  Milestones;  Program  Acquisition 
Events;  Contract  Release  and  Awards;  Program  Deliverables; 
Developmental  Test  and  Evaluation;  Operational  Test  and 
Evaluation  and  Program  Funding.  Figure  3-9  provides  an 
illustrated  example  of  an  integrated  test  program  schedule. 

For  ACAT  III  and  IV  programs  not  on  the  OSD  T&E  Oversight 
list  it  is  not  critical  to  adhere  to  the  exact  format  of  figure 
3-9.  A  chart  showing  the  program  milestones  and  the  planned 
tests  is  adequate. 

Can  be  a  fold-out 

Must  cover  the  acquisition  and  test  &  evaluation  program 
through  full  operational  capability 

The  integrated  time  sequencing  of  critical  events  will  be 
reflected  as  appropriate  (for  example) : 

MILESTONES:  I,  II,  III,  First  Unit  Equiped  (FUE) ,  Initial 
Operational  Capability  (IOC) . 

FORMAL  SOLICITATION  RELEASE: 

Demonstration  Validation  (Dem-Val)  RFP  Release 
Low  Rate  Initial  Production  (LRIP)  RFP  Release 
Engineering  &  Manufacturing  Development  RFP  Release 
Full  Rate  Production  (FRP)  Long  Lead  RFP  Release 


CONTRACT  AWARD  OR  EVENT: 

Demonstration  Validation  Award 

Engineering  &  Manufacturing  Development  Award 

LRIP  Long  Lead  Item  Award 

LRIP  Options 

FRP  Long  Lead  Award 

Preliminary  Design  Review  (PDR) 

Critical  Design  Review  (CDR) 

DELIVERIES: 

Brassboard 

Prototype  (Designate  Quantity) 

LRIP  (Designate  Quantity) 

Production  (Designate  Quantity) 

TECHNICAL  TEST  &  EVALUATION  (DT&E) : 

Developmental  Tests _ 
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Pre-Milestone  II 

Technical  Feasibility  Test  (TFT) 

Engineering  Development  Test  (EDT) 

Pre-Milestone  III 

Pre-production  Qualification  Test  (PPQT) 

Production  Prove-Out  Test  (PPT) 

Live  Fire  Test  (LFT) 

Logistic  Demonstration  (LD) 

Production  and  Deployment  Phase 

Production  Qualification  Test  (PQT) 

First  Article  Test  (FAT) 

USER  TEST  &  EVALUATION  (OT&E) : 

Early  Operational  Assessment 
Operational  Assessment 

Operational  Tests 

Pre-MS  II 

Early  User  Test  (EUT) 

Early  User  Experiment  (EUE) 

Pre-MS  III 

Limited  User  Test  (LUT) 

Initial  Operational  Test  (lOT) 

Production  and  Deployment  Phase 

Follow-on  Operational  Test  (FOT) 

As  Required 

Force  Development  Test  (FDT) 

Force  Development  Experiment  (FDE) 

Concept  Evaluation  Program  Test  (CEP) 

FUNDING  -  cumulative  by  fiscal  year 

-  include  all  funds  expended  by  the  PM,  support 
agencies  and  the  test  agencies. 

MRTFB  Reimbursable  -  obtain  data  from  the  program  planning 
forecast  document  that  addresses  developmental  test  at  US 
Army  Test  and  Evaluation  Command  (TECOM)  test  facilities  and 
other  DoD  managed  facilities.  See  Chapter  8,  AR  73-1  and 
Part  Four . 

RDT&E  -  include  all  RDT&E  expenditures,  not  just  T&E  related 

-  include  DT&E  and  OT&E  costs 

-  include  LRIP  and  Test  Articles  for  DT&E  and  lOT&E 

-  as  described  in  Chapter  8,  AR  73-1 
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PROCUREMENT  -  as  described  in  program  baseline 

For  ACAT  III  and  IV  programs,  not  designated  for  OSD  T&E 
oversight,  funding  information  on  the  Integrated  Program 
Schedule  is  optional. 


b. 


(1)  Discuss  the  test  and  evaluation  responsibility  of  all 
participating  organizations  (developers,  testers,  evaluators, 
users) . 


Identify  TIWG  members  and  their 

role.  Reference  the  TIWG 

charter  for  specific  responsibilities.  (See  AR  73“1  and  Part  1 

One,  Chapter  8.)  The  TIWG  charter 

must  be  included  as  a  I 

reference  in  Appendix  A,  Bibliography.  For  example:  1 

Program  Manager/TIWG  Chair 

PM  (System) 

Combat  Developer 

TRADOC  proponent  school 

Operational  Evaluator 

OEC 

Operational  Tester 

TEXCOM 

Developmental  Evaluator /Assessor 

AMSAA/TECOM 

Developmental  Tester 

TECOM 

Logistician 

AMSAA 

Sur vi vabi lity/Lethality 

SLAD,  Army  Research  Lab 

Threat  Integrator 

DCSINT 

Participating  Service  OTA 
Participating  Service  User 

AFOTEC,  MCOTEA,  COMOPTEVFOR 

Representative 
(if  multi-service) 

IV&V  Agency 

ARDEC 

For  ACAT  III  and  IV  programs,  not  designated  for  OSD  T&E 
oversight,  it  is  sufficient  to  reference  the  TIWG  charter. 

If  the  Human  Use  Committee  (HUC) 

makes  a  recommendation  that  I 

there  is  no  further  test  plan  review  required  and  that 
recommendation  is  approved  by  the  test  plan  approval  authority, 
the  recommendation  is  to  be  noted  in  this  paragraph  and 
reference  made  to  the  decision  document  in  Appendix  A, 

Bibliography.  (See  AR  70-25) 

(2)  Provide  the  date  (fiscal  quarter)  when  the  decision  to 
proceed  beyond  low-rate  initial  production  is  planned.  (Low-rate 
initial  production  quantities  required  for  operational  test  must 
be  identified  for  the  Director  of  Operational  Test  and  Evaluation 
approval  prior  to  Milestone  II  for  acquisition  category  I  programs 
and  other  acquisition  category  programs  designated  for  Office  of 
the  Secretary  of  Defense  test  and  evaluation  oversight) . 
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The  date  for  the  Beyond  Low  Rate  Initial  Production  (BLRIP) 
decision  is  found  in  the  Integrated  Program  Summary  (IPS), 
Acquisition  Strategy  Report. 

The  quantity  of  LRIP  items  needed  for  lOT  is  recommended  by 
OPTEC  in  coordination  with  the  program  manager  and  included  for 
approval  by  DOT&E  for  ACAT  I  and  other  ACAT  programs  having  OSD 
test  and  evaluation  oversight. 

The  quantity  of  items  needed  for  lOT  for  all  other  ACAT 
programs  are  included  as  recommended  by  OPTEC. 


(3)  Identify  and  discuss  any  operational  issues  and 
vulnerability  and  lethality  Live  Fire  Test  requirements  that  will 
not  be  addressed  before  proceeding  beyond  low-rate  initial 
production. 

PART  III  —  DEVELOPMENTAL  TEST  AND  EVALUATION  QUTLIIffi 

a.  Developmental  Test  and  Evaluation  Overview.  Explain  how 
developmental  test  and  evaluation  will:  verify  the  status  of 
engineering  and  manufacturing  development  progress;  verify  that 
design  risks  have  been  minimized;  and  substantiate  achievement  of 
contract  technical  performance  requirements;  and  be  used  to 
certify  readiness  for  dedicated  operational  test.  Specifically, 
identify: 

(1)  Any  technology /subsystem  that  has  not  demonstrated  its 
ability  to  contribute  to  system  performance  and  ultimately  fulfill 
mission  requirements. 

(2)  The  degree  to  which  system  hardware  and  software  design 
has  stabilized  so  as  to  reduce  manufacturing  and  production 
decision  uncertainties. 


Summarize  the  entire  developmental  test  and  evaluation 
program. 

Present  a  narrative  walk-through  of  the  integrated  schedule, 
discussing  the  interrelationships  between  tests,  developmental 
and  operational,  and  between  tests  and  milestones.  Do  not 
duplicate  details  that  will  be  found  in  para  3c,  Future 
Developmental  Test  and  Evaluation.  The  purpose  of  the  Overview 
is  to  identify  how  the  individual  tests  fit  within  the  framework 
of  the  overall  program  and  the  continuous  evaluation  process. 
Some  of  the  topics  that  need  to  be  addressed  in  this  paragraph 
include: 
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(1)  Early  develop»mental  tests  that  will  be  performed  to 
mitigate  technical  risks  in  the  program  that  are  defined  in  the 
Risk  Assessment,  Annex  D,  Integrated  Program  Summary  (Reference 
DoD  5000. 2-M,  Part  4,  Annex  E.). 

(2)  Identification  of  developmental  tests  that  will  be  used 
to  demonstrate  that  the  test  item  is  safe,  that  the  technical 
manuals  are  verified  and  validated  and  ready  for  usage  in  a 
following  or  concurrent  Operational  Test. 

(3)  Identification  of  the  test,  usually  Pre-production 
Qualification  Test  (PPQT) ,  that  will  be  performed  to  validate 
that  the  system  meets  the  program's  technical  performance 
requirements  that  are  usually  contractually  mandated  in  a 
specification. 

(4)  The  developmental  test(s)  that  will  be  used  to  certify 
the  system  is  ready  for  Initial  Operational  Test  (lOT)  and  who 
has  responsibility  for  execution. 

(5)  If  applicable,  testing  to  address  conventional  weapon 
effects.  Electromagnetic  and  Environmental  Effect  (E^), 

ECM/ECCM,  initial  nuclear  weapons  effects,  advanced  technology 
survivability  and  NBC  contamination  survivability  (Reference 
DoDI  5000.2,  Part  6,  Section  F.) 

(6)  Identification  of  the  test  plans  and  strategy  to  prove 
or  validate  the  manufacturing  process  (Reference  DoDI  5000.2, 
Part  6,  Section  0). 

The  following  areas  need  to  be  addressed  throughout 
Developmental  Test  and  Evaluation  (They  are  addressed  in  general 
in  the  DT&E  Overview  and  specifically  in  the  description, 
objective,  etc,  of  each  of  the  developmental  tests  addressed  in 
Future  DT&E.): 

Reliability,  Availability,  and  Maintainability  (Reference 
DoDI  5000.2,  Part  6,  Section  C.) 

Electromagnetic  Compatibility  and  Radio  Frequency  Management 
(Reference  DoDI  5000.2,  Part  6,  Section  G) 

Human  Factors  (Reference  DoDI  5000.2,  Part  6,  Section  H) 

System  Safety,  Health  Hazards  and  Environment  (Reference  DoDI 
5000.2,  Part  6,  Section  I) 
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Integrated  Logistical  Support  (Reference  DoDI  5000.2,  Part  7, 
Section  A) .  A  Logistics  Demonstration  (LD)  is  required  for  all 
acquisition  programs  unless  waived.  (See  AR  700-127)  The 
waiver  if  approved  will  be  documented  in  Part  II,  paragraph  2 
with  the  approval  document  referenced  in  the  Bibliography, 
Appendix  A. 

Discuss  the  indicators  that  will  be  used  to  determine 
software  status  and  evaluate  progress  toward  software  maturity 
in  support  of  key  decision  points,  particularly  for  software 
intensive  systems.  Show  how  the  indicators  in  each  phase  relate 
to  those  in  previous  and  subsequent  phases. 


b.  Developmental  Test  and  Evaluation  to  Date.  Identify 
completed  developmental  test  and  evaluation  by  noting  on  the 
matrix  of  critical  technical  parameters  those  parameters  that  have 
been  demonstrated. 


Update  the  Critical  Technical  Parameters  matrix  in  Part  I. 
Note  the  actual  values  that  have  been  demonstrated. 

For  parameters  not  met,  provide  a  brief  explanation  as  to  why 
and  performance  impact.  Identify  future  test  that  will  re¬ 
address  parameters. 

A  detailed  discussion  of  the  results  of  testing  is  not 
required. 

Test  and  evaluation  reports  prepared  to  date  must  be  included 
as  references  in  Appendix  A.  Bibliography. 


c.  Future  Developmental  Test  and  Evaluation.  Discuss  all 
remaining  developmental  test  and  evaluation  that  is  planned, 
beginning  with  the  date  of  the  current  Test  and  Evaluation  Master 
Plan  revision  and  extending  through  completion  of  production. 
Place  emphasis  on  the  next  phase  of  testing.  For  each  phase, 
include: 


For  each  test  within  each  remaining  acquisition  phase  address 
the  following  items:  configuration  description,  DT&E 
objectives,  DT&E  events,  scope,  basic  scenarios  and  limitations 
e.g.,  (1)  Demonstration  Validation 
(a)  Chassis  Design  Test 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing,  and  Basic  Scenarios 

(4)  Limitations 
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(2)  Engineering  and  Manufacturing  Development 
(a)  Pre-Production  Qualification  Test 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testings  and  Basic  Scenarios 

(4)  Limitations 

For  those  critical  technical  parameters  where  demonstrated 
value  did  not  meet  the  threshold  or  objective,  planned  testing 
must  insure  that  these  parameters  will  be  re-addressed. 


(1)  Configuration  Description.  Summarize  the  functional 
capabilities  of  the  system's  developmental  configuration  and  how 
they  differ  from  the  production  model. 


List  the  difference  between  the  system  to  be  tested  and  the 
objective  system,  to  include  software. 


(2)  Developmental  Test  and  Evaluation  Ob-iectives.  State  the 
test  objectives  for  this  phase  in  terms  of  the  critical  technical 
parameters  to  be  confirmed.  Identify  any  specific  technical 
parameters  which  the  milestone  decision  authority  has  designated 
as  exit  criteria  and/or  directed  to  be  demonstrated  in  a  given 
phase  of  testing. 


Exit  criteria  are  generally  found  in  the  Acquisition  Decision 
Memorandum  (ADM)  for  ACAT  I  &  II  programs. 

For  ACAT  III  &  IV,  exit  criteria  can  be  found  in  the  IPR 
decision  docxamentation. 


(3)  Developmental  Test  and  Evaluation  Events.  Scope  of 
Testing,  and  Basic  Scenarios.  Summarize  the  test  events,  test 
scenarios  and  the  test  design  concept  Quantify  the  testing  (i.e., 
number  of  test  hours,  test  events,  test  firings).  List  the 
specific  threat  systems,  surrogates,  countermeasures,  component  or 
subsystem  testing,  and  testbeds  the  use  of  which  are  critical  to 
determine  whether  developmental  test  objectives  are  achieved.  As 
appropriate,  particularly  if  an  agency  separate  from  the  test 
agency  will  be  doing  a  significant  part  of  the  evaluation, 
described  the  methods  of  evaluation.  List  all  models  and 
simulations  to  be  used  and  explain  the  rationale  for  their 
credible  use.  Describe  how  performance  in  natural  environmental 
conditions  representative  of  the  intended  area  of  operations 
(e.g.,  temperature,  pressure,  humidity,  fog,  precipitation, 
clouds,  blowing  dust  and  sand,  icing,  wind  conditions,  steep 
terrain,  wet  soil  conditions,  high  sea  state,  storm  surge  and 
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tides,  etc.)  and  interoperability  and  compatibility  with  other 
weapon  and  support  systems  as  applicable  will  be  tested. 


The  resources  identified  must  correspond  to  those  listed  in 
Part  V. 

Include  discussion  of  any  test  databases  and/or  remote 
terminal  emulators  to  be  used  and  their  relationship  to  the 
objective  system  environment. 


(4)  Limitations.  Discuss  the  test  limitations  that  may 
significantly  affect  the  evaluator's  ability  to  draw  conclusions, 
the  impact  of  these  limitations,  and  resolution  approaches. 


Identify  the  differences  between  the  COEA  environment  and  the 
test  environment  that  would  affect  the  ability  to  use  test  data  in 
validating  the  COEA  database  used  for  predicting  operational 
effectiveness . 


d.  Live  Fire  Test  and  Evaluation.  Include  a  description  of 
the  overall  live  fire  test  and  evaluation  strategy  for  the  item; 
critical  live  fire  test  and  evaluation  issues;  required  levels  of 
system  vulnerability/lethality;  the  management  of  the  live  fire 
test  and  evaluation  program  live  fire  test  and  evaluation 
schedule,  funding  plans  and  requirements;  related  prior  and  future 
live  fire  test  and  evaluation  efforts;  the  evaluation  plan  and 
shot  selection  process;  and  major  test  limitations  for  the  conduct 
of  live  fire  test  and  evaluation.  Live  fire  test  and  evaluation 
resource  requirements  (including  test  articles  and 
instrumentation)  will  be  appropriately  identified  in  the  Test  and 
Evaluation  Resource  Summary. 


This  paragraph  applies  to  those  systems  that  are  identified 
as  a  covered  system  or  major  munition  program  as  defined  in 
Title  10,  United  States  Code,  Section  2366. 

Do  not  address  LFT&E  in  a  separate  annex. 

See  Part  Five  for  details  on  LFT&E. 

Group  all  vulnerability/lethality  testing  (when  applicable) 
under  one  paragraph  to  show  how  the  vulnerability/lethality 
issue  is  being  assessed  through  various  tests  and  subtests. 

Such  testing  can  include  dedicated  tests  such  as  Ballistic  Hull 
and  Turret  testing  and  Live  Fire  Test.  Subtests  can  include 
armor  plate  tests,  penetration  tests,  as  well  as  other  tests 
that  validate  the  vulnerability/lethality  requirements  of  a 
program. 
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Provide  an  executive  level  summary  discussion. 

Summarize  LFT  details  as  appropriate  throughout  the  TEMP. 
Leave  detailed  discussion  to  the  test  plans. 


(1)  The  primary  purpose  of  operational  testing  and 
evaluation  is  to  verify  that  operationally  effective  and 
operationally  suitable  systems  are  approved  for  production  that 
meet  the  mission  needs  and  minimum  operational  performance 
requirements  of  the  operating  forces. 

(2)  The  Test  and  Evaluation  Master  Plan  will  show  how 
program  schedule,  test  management  structure,  and  required 
resources  are  related  to  operational  requirements,  critical 
operational  issues,  test  objectives,  and  milestone  decision 
points.  Testing  will  evaluate  the  system  (operated  by  typical 
users)  in  an  environment  as  operationally  realistic  as  possible, 
including  threat  representative  hostile  forces  and  the  expected 
range  of  natural  environmental  conditions. 


Summarize  the  entire  operational  test  and  evaluation  program 
and  the  evaluation  strategy.  Present  a  narrative  walJc-through 
of  the  integrated  schedule  discussing  the  interrelationships 
between  contractor,  government,  developmental  and  operational 
tests,  models  and  simulations  and  the  milestones  they  support. 

Do  not  duplicate  the  details  that  are  provided  in  paragraph  4.d, 
Future  Operational  Test  and  Evaluation.  The  purpose  of  the 
overview  is  to  give  a  quick,  concise  look  at  the  overall  test 
program,  explaining  the  many  interrelationships  and 
opportunities  to  conduct  continuous  evaluation  (CE) .  Some  of 
the  topics  that  need  to  be  addressed  include: 

(1)  Identification  of  contractor  and  developmental  tests 
that  will  be  used  as  part  of  an  operational  evaluation  or 
assessment. 

(2)  Identification  of  simulations  that  will  be  used  to 
augment  and  extend  operational  testing  as  part  of  an  operational 
evaluation  or  assessment. 

(3)  Key  characteristics  of  the  system  that  will  be  the  focus 
of  the  evaluation. 

(4)  Sources  of  data,  baseline  comparisons,  general  analysis 

scheme  and  test  data/COEA  linkage. _ _ 
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The  following  areas  need  to  be  addressed  throughout 
Operational  Test  and  Evaluation  (They  are  addressed  in  general 
in  the  OT&E  Overview  and  specifically  in  the  description, 
objective,  etc,  of  each  of  the  operational  tests  addressed  in 
Future  OT&E.): 

Human  performance  issues  must  be  addressed  (Reference  DoDI 
5000.2,  part  7,  Section  B) . 

Logistics  support  issues  (readiness,  reliability, 
availability  and  maintainability)  to  include  Test,  Measurement 
and  Diagnostic  Equipment  (TMDE)  and  integrated  diagnostics  must 
be  addressed  (Reference  DoDI  5000.2,  part  7,  Section  A). 


b.  Critical  Operational  Issues 

(1)  List  in  this  section  the  critical  operational  issues. 
Critical  operational  issues  are  the  operational  effectiveness  and 
operational  suitability  issues  (not  parameters,  objectives  or 
thresholds)  that  must  be  examined  in  operational  test  and 
evaluation  to  evaluate/assess  the  system's  capability  to  perform 
its  mission. 

(2)  A  critical  operational  issue  is  typically  phrased  as  a 
question  that  must  be  answered  in  order  to  properly  evaluate 
operational  effectiveness  (e.g.,''Will  the  system  detect  the  threat 
in  a  combat  environment  at  adequate  range  to  allow  successful 
engagement?")  and  operational  suitability  (e.g.,  "Will  the  system 
be  safe  to  operate  in  a  combat  environment?"). 

(3)  Some  critical  operational  issues  will  have  critical 
technical  parameters  and  minimum  acceptable  operational 
performance  requirements  or  thresholds.  Individual  attainment  of 
these  attributes  does  not  guarantee  that  the  critical  operational 
issue  will  be  favorably  resolved.  The  judgment  of  the  operational 
test  agency  is  used  by  the  DoD  Component  to  determine  if  the 
critical  operational  issue  is  favorably  resolved. 

(4)  If  every  critical  operational  issue  is  resolved 
favorably,  the  system  should  be  operationally  effective  and 
operationally  suitable  when  employed  in  its  intended  environment 
by  typical  users. 


TRADOC  approved  Critical  Operational  Issues  and  Criteria 
(COIC)  are  required  for  all  programs  at  MS  I  and  for  ACAT  III  & 
IV  programs  at  all  milestones. 

DCSOPS  approved  COIC  are  required  for  ACAT  I,  li  and  OSD  T&E 
oversight  systems  at  MS  II  and  beyond. 
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Pixmy  policy  (AR  73-1,  para  5-8)  requires  approved  COIC  be 
included  in  the  TEMP. 

Include  the  approved  COICs  in  their  entirety  in  the  TEMP; 
this  includes  Issue,  Scope,  Criteria  and  Rationale. 

Discuss  the  relationships  between  the  criteria  in  the  COIC, 
the  minimum  acceptable  operational  performance  requirements  in 
the  ORD  and  the  MOEs  with  supporting  MOPs  in  the  COEA.  Should 
be  part  of  the  COIC  rationale  statement. 

Reference  the  COIC  approval  document  in  Appendix  A, 
Bibliography. 

c.  Operational  Test  and  Evaluation  to  Date.  Identify  and 
date  test  reports  that  detail  the  results  of  testing  and 
operational  assessments  to  date.  Indicate  critical  operational 
issues  that  were  resolved  (satisfactory,  unsatisfactory,  yes,  no 
etc.),  partially  resolved,  or  unresolved  at  the  completion  of  each 
phase  of  testing. 

The  results  related  to  the  resolution  of  the  criteria  in 
addition  to  the  overall  issue  should  be  discussed. 

Ensure  that  all  test  reports  referenced  are  listed  in 
Appendix  A,  Bibliography.  Reports  must  be  available  if 
requested. 

For  software,  within  the  context  of  previously  identified 
operational  issues,  summarize  what  has  been  learned  from  the 
operational  test  accomplished  to  date  about  the  maturity  of  the 
software.  Show  how  operational  test  results  from  interim 
hardware  and  software  configurations  apply  to  configurations 
intended  for  deployment.  Identify  differences  between  tested 
software,  software  planned  for  the  current  phase,  and  software 
to  be  deployed.  Discuss  the  importance  of  these  differences. 


d.  Future  Operational  Test  and  Evaluation.  For  each 
remaining  phase  of  operational  test  and  evaluation,  separately 
address  the  following: 


Identify  operational  tests  that  will  be  conducted  and  the 
developmental  tests  that  will  be  used  as  data  for  operational 
evaluation  or  assessment.  When  developmental  tests  are 
identified,  subparagraph  (3)  Events,  Scope  of  Testing  and 
Scenarios  should  define  the  data  in  general  terms  that  will  be 
taken  from  the  developmental  test  for  the  evaluation  or 
assessment.  This  will  ensure  that  the  developmental  testers  and 
evaluators,  by  their  signature  on  the  TEMP,  have  agreed  to 
collect  and  provide  that  data  to  the  operational  evaluator. 
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Describe  how  models  will  be  accredited  for  use  in  this 
specific  application.  The  approval  vehicle  for  accreditation  is 
an  Accreditation  Plan  as  outlined  in  DUSA(OR)  memo  dated  30  Oct 
89,  subject:  Verification,  Validation,  and  Accreditation  of 
Models.  Reference  the  Accreditation  Plan  in  Appendix  A, 
Bibliography. 

Part  V,  Resource  Summary  will  identify  the  resources 
necessary  to  perform  the  validation  and/or  accreditation. 


If  more  than  one  test  is  in  a  phase,  the  following  layout 
should  be  included  for  each  test.  For  example: 

(1)  Demonstration/Validation 

(a)  Early  User  Test  (EUT) 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing  and  Scenarios 

(4)  Limitations 

(b)  (Next  test  within  Dem/Val  phase) 

(Note:  Either  list  each  sub-element  for  the  developmental  test  to 
be  used  for  data  or  refer  to  the  applicable  paragraph  in  Part 
III  that  contains  the  information.) 

(2)  Engineering  and  Manufacturing  Development 

(a)  Initial  Operational  Test  and  Evaluation 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing  and  Scenarios 

(4)  Limitations. 

(b)  (Next  test  within  Eip  phase.) 


(1)  Configuration  Description.  Identify  the  system  to  be 
tested  during  each  phase,  and  describe  any  differences  between  the 
tested  system  and  the  system  that  will  be  fielded  including,  where 
applicable,  software  maturity  performance  and  criticality  to 
mission  performance,  and  the  extent  of  integration  with  other 
systems  with  which  it  must  be  interoperable  or  compatible. 
Characterize  the  system  (e.g.,  prototype,  engineering  development 
model,  production  representative  or  production  configuration) 

(2)  Operational  Test  and  Evaluation  Ob-i actives.  State  the 
test  objectives  including  the  minimum  acceptable  operational 
performance  requirements  and  critical  operational  issues  to  be 
addressed  by  each  phase  of  operational  test  and  evaluation  and  the 
milestone  decision  review (s)  supported.  Operational  test  and 
evaluation  that  supports  the  beyond  low  rate  initial  production 
decision  should  have  test  objectives  that  examine  all  areas  of 
operational  effectiveness  and  suitability. 

r 
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Discuss  the  relationship  between  OT&E  objectives  and  the 
software  characteristics  which  affect  critical  operational 
issues. 

For  FOT&E,  identify  major  deficiency  corrections  to  be 
verified.  OTs  should  be  designed  to  assure  that  software  is 
fault  tolerant  and  supportable. 


(3)  Operational  Test  and  Evaluation  Events.  Scope . of 

Testing,  and  Scenarios.  Summarize  the  scenarios  and  identify  the 
events  to  be  conducted,  type  of  resources  to  be  used,  the  threat 
simulators  and  the  simulation (s)  to  be  employed,  the  type  of 
representative  personnel  who  will  operate  and  maintain  the  system, 
the  status  of  the  logistic  support,  the  operational  and 
maintenance  documentation  that  will  be  used,  the  environment  under 
which  the  system  is  to  be  employed  and  supported  during  testing, 
the  plans  for  interoperability  and  compatibility  testing  with 
other  United  States/Allied  weapon  and  support  systems  as 
applicable,  etc.  Identify  planned  sources  of  information  (e.g., 
developmental  testing,  testing  of)  related  systems,  modeling, 
simulation,  etc.)  that  may  be  used  by  the  operational  test  agency 
to  supplement  this  phase  of  operational  test  and  evaluation. 
Whenever  models  and  simulations  are  to  be  used,  explain  the 
rationale  for  their  credible  use.  If  operational  test  and 
evaluation  cannot  be  conducted  or  completed  in  this  phase  of 
testing  and  the  outcome  will  be  an  operational  assessment  instead 
of  an  evaluation,  this  should  clearly  be  stated  and  the  reason (s) 
explained. 


Include  a  description  of  the  relationship  between  software 
functions  being  tested  and  test  scenario  events  that  will  cause 
that  function  to  be  exercised.  Identify  load  levels  to  be  used 
and  their  relationship  to  the  required  operational  environment. 


(4)  Limitations.  Discuss  the  test  limitations  including 
threat  realism,  resource  availability,  limited  operational 
(military,  climatic,  nuclear,  etc.)  environments,  limited  support 
environment,  maturity  of  tested  system,  safety,  etc.,  that  may 
impact  the  resolution  of  affected  critical  operational  issues. 
Indicate  the  impact  of  the  test  limitations  on  the  ability  to 
resolve  critical  operational  issues  and  the  ability  to  formulate 
conclusions  regarding  operational  effectiveness  and  operational 
suitability.  Indicate  the  critical  operational  issues  affected  in 
parenthesis  after  each  limitation. 
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Identify  any  factors  which  may  inhibit  realistic  OT  of  the 
software.  Constraints  imposed  by  software  maturity  or 
availability  of  resources  and  simulators  should  be  given  along 
with  their  impact  on  critical  operational  issues. 

Identify  differences  between  the  COEA  environment  and  the 
test  environment  that  would  affect  the  ability  to  use  test  data 
in  validating  the  COEA  database  used  for  predicting  operational 
effectiveness. 


FART  V  —  TEST  AND  EVALUATION  RESOURCE  SUMMARY 

Provide  a  summary  (preferably  in  a  table  or  matrix  format)  of 
all  key  test  and  evaluation  resources,  both  government  and 
contractor,  which  will  be  used  during  the  course  of  the 
acquisition  program.  Specifically,  identify  the  following  test 
resources : 


A  matrix  or  table  is  preferred  to  identify  planned  use  of 
available  key  assets  and  requirements  for  a  new  or  unique 
capability  or  item  that  needs  to  be  acquired  or  developed  to 
support  the  test  program.  The  following  should  be  addressed: 

test  articles 

test  sites  and  instrumentation 
test  support  equipment 
threat  systems /simulators 
test  targets  and  expendables 
simulations,  models  and  testbeds 

Developmental  tester  and  operational  tester  should  provide 
input  specific  to  their  requirements.  Indicate  which 
requirements  were  identified  by  each  tester 

Existing  capabilities  that  are  key  to  accomplishing  the  test 
program  need  be  included  and  specifically  all  those  where  use  is 
known  to  be  restricted  or  significant  upgrade  or  improvement  is 
needed. 

The  matrix  should,  at  a  minimum,  identify  the  item,  the 
quantity  or  number  required,  the  location,  the  test  event  or 
time  frame  when  needed,  the  resources  required  to  be  obtained 
and  the  organization/activity  responsible  for  acquisj.t  ion  or 
development . 

Software  resource  requirements  are  found  in  the  Computer 
Resources  Life  Cycle  Management  Plan  (CRLCMP) . 
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a.  TPQ^  Artirles.  Identify  the  actual  number  of  and  timing 
requirements  for  all  test  articles,  including  key  support 
equipment  and  technical  information  required  for  testing  in  each 
phase  by  major  type  of  developmental  test  and  evaluation  and 
operational  test  and  evaluation.  If  key  subsystems  (components, 
assemblies,  subassemblies  or  software  modules)  are  to  be  tested 
individually,  before  being  tested  in  the  final  system 
configuration,  identify  each  subsystem  in  the  Test  and  Evaluation 
Master  Plan  and  the  quantity  required.  Specifically  identify  when 
prototype,  engineering  development,  pre-production,  or  production 
models  will  be  used. 

b.  Test  Sites  and  Instrumentation.  Identify  the  specific 
test  ranges/facilities  to  be  used  for  each  type  of  testing. 

Compare  the  requirements  for  test  ranges/facilities  dictated  by 
the  scope  and  content  of  planned  testing  with  existing  and 
programmed  test  range/facility  capability,  and  highlight  any  major 
shortfalls,  such  as  inability  to  test  under  representative  natural 

environmental  conditions .  _ 

Identify  instrumentation  that  must  be  acquired  specifically 
to  conduct  the  planned  test  program. 


This  includes  software  facilities  and  tools  to  support 
testing  identified  in  Parts  III  and  IV. 

Address  shortfalls  and  impact  under  Limitations  in  Part  III 
and/or  Part  IV  as  applicable. 

Testing  shall  be  planned  and  conducted  to  take  full  advantage 
of  existing  investment  in  DOD  ranges,  facilities  and  other 
resources,  wherever  practical  (Reference  DoDI  5000.2,  Part  8, 
paragraph  2 . d . ( 4 ) ) . 

In  order  for  the  Army  to  realize  maximum  value  of  its  capital 
investment  in  test  facilities,  it  is  necessary  that  PEO/PMs 
coordinate  developmental  test  and  evaluation  requirements  with 
TECOM.  This  should  be  accomplished  early  in  the  acquisition 
cycle,  preferably  prior  to  MS  I.  This  coordination  should 
facilitate  the  development  of  developmental  testing  requirements 
and  determine  the  extent  and  nature  of  contractor  services,  if 
required.  If  TECOM  cannot  conduct  the  developmental  test  (e.g., 
scheduling  does  not  permit) ,  the  PEO/PM  has  the  authority  to  use 
contractor  support.  This  decision  and  rationale  will  be 
documented  in  this  paragraph  of  the  TEMP. 

Address  instrumentation  that  must  be  developed  and/or 
procured.  Clearly  identify  the  test  investment  requirement  to 
ensure  test  site  instrumentation  availability  and  capability. 


c,  Tpst  Snnnort  F.rniipment.  Identify  test  support  equipment 
that  must  be  acquired  specifically  to  conduct  the  test  program. 
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Only  address  new  test  support  equipment.  This  includes 
software  test  drivers,  emulators  or  diagnostics,  if  applicable, 
to  support  identified  testing.  Identify  unique  or  special 
calibration  requirements  associated  with  this  test  support 
equipment . 


d.  Threat  Systems /Simulators.  Identify  the  type,  number, 
availability,  and  fidelity  requirements  for  all  threat 
systems/simulators.  Compare  the  requirements  for  threat 
systems/simulators  with  available  and  projected  assets  and  their 
capabilities.  Highlight  any  major  shortfalls.  Each  threat 
simulator  shall  be  subjected  to  validation  procedures  to  establish 
and  document  a  baseline  comparison  with  its  associated  threat  and 
to  ascertain  the  extent  of  the  operational  and  technical 
performance  differences  between  the  two  throughout  the  simulator's 
life-cycle. 


Threat  systems/simulators  to  be  used  in  activities  supporting 
milestone  decisions  must  be  validated  and  accredited  for  the 
specific  application.  Validation  and  accreditation  procedures 
are  to  be  documented  in  accordance  with  the  Army  Validation  and 
Accreditation  Plan  as  described  in  Part  Nine.  The  resulting 
report  should  be  cited  in  Appendix  A,  Bibliography. 


e.  Test  Targets  and  Expendables.  Identify  the  type,  number, 
and  availability  requirements  for  all  targets,  flares,  chaff, 
sonobuoys,  smoke  generators,  acoustic  countermeasures,  etc.,  that 
will  be  required  for  each  phase  of  testing.  Identify  any  major 
shortfalls. 


Include  threat  targets  for  LFT  lethality  testing  and  threat 
munitions  for  vulnerability  testing.  High  fidelity  targets 
require  the  same  validation  and  accreditation  process  as  for 
threat  systems  and  simulators.  Procedures  described  in  Part 
Nine  also  apply.  Results  of  this  effort  should  be  cited  in 
Appendix  A,  Bibliography. 


f.  Operational  Force  Test  Support.  For  each  test  and 
evaluation  phase,  identify  the  type  and  timing  of  aircraft  flying 
hours,  ship  steaming  days,  and  on-orbit  satellite 
contacts/coverage,  and  other  critical  operating  force  support 
required. 
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Include  size,  location  and  type  unit  of  unit  required. 


g.  Simulation.  Models,  and  Testbeds.  For  each  test  and 
evaluation  phase,  identify  the  system  simulations  required, 
including  computer-driven  simulation  models  and  hardware/software- 
in-the-loop  testbeds.  Identify  the  resources  required  to  validate 
and  certify  their  credible  usage  or  application  before  their  use. 


Include  only  those  simulations,  models  and  testbeds  that  will 
be  used  to  extend  testing  and/or  used  in  evaluation.  This 
includes  feeder  models. 

Simulations,  models  and  test  beds  used  solely  for  engineering 
purposes  (not  in  support  of  program  decisions)  do  not  need  to  be 
identified  in  this  paragraph.  The  items  identified  in  this 
paragraph  should  have  an  accreditation  plan  developed  as  outlined 
in  DUSA(OR)  memorandum,  30  Oct  89,  Subject:  Verification, 
Validation  and  Accreditation  of  Models. 


h.  Special  Requirements.  Discuss  requirements  for  any 
significant  non- instrumentation  capabilities  and  resources  such 
as:  special  data  processing/data  bases,  unique 

mapping /charting /geodesy  products,  extreme  physical  environmental 
conditions  or  restricted/special  use  air/sea/landscapes. 

i.  Test  and  Evaluation  Funding  Requirements.  Estimate,  by 
fiscal  year  and  appropriation  line  number  (program  element),  the 
funding  required  to  pay  direct  costs  of  planned  testing.  State, 
by  fiscal  year,  the  funding  currently  appearing  in  those  lines 
(program  elements) .  Identify  any  major  shortfalls. 


Use  of  a  table  or  matrix  is  preferred. 
Show  potential  shortfalls. 


j.  Manpower /Personnel  Training.  Identify  manpower /personnel 
and  training  requirements  and  limitations  that  affect  test  and 
evaluation  execution. 


The  preliminary  Test  and  Evaluation  Master  Plan  should 
project  the  key  resources  necessary  to  accomplish  demonstration 
and  validation  testing  and  early  operational  assessment.  The 
preliminary  Test  and  Evaluation  Master  Plan  should  estimate,  to 
the  degree  known  at  Milestone  I,  the  key  resources  necessary  to 
accomplish  developmental  test  and  evaluation,  live  fire  test  and 
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evaluation,  and  operational  test  and  evaluation.  These  should 
include  elements  of  the  National  Test  Facilities  Base  (which 
incorporates  the  Major  Range  and  Test  Facility  Base  (MRTFB) , 
capabilities  designated  by  industry  and  academia,  and  Major  Range 
and  Test  Facility  Base  test  equipment  and  facilities) ,  unique 
instrumentation,  threat  simulators,  and  targets.  As  system 
acquisition  progresses,  the  preliminary  test  resource  requirements 
shall  be  reassessed  and  refined  and  subsequent  Test  and  Evaluation 
Master  Plan  updates  shall  reflect  any  changed  system  concepts, 
resource  requirements,  or  updated  threat  assessments.  Any 
resource  shortfalls  which  introduce  significant  test  limitations 
should  be  discussed  with  planned  corrective  action  outlined. 


This  paragraph  contains  overall  guidance  for  preparing  a 
preliminary  TEMP,  i.e.,  a  TEMP  to  support  Milestone  I.  It  is 
not  a  separate  paragraph  to  be  addressed  in  the  TEMP. 


APPENDIX  A  —  BIBLIOGRAPHY 

a.  Cite  in  this  section  all  documents  referred  to  in  the 
Test  and  Evaluation  Master  Plan. 

b.  Cite  all  reports  documenting  developmental  and 
operational  testing  and  evaluation. 

APPENDIX  B  --  ACRONYMS.  List  and  define  all  acronyms  used  in 
the  Test  and  Evaluation  Master  Plan. 

APPENDIX  C  --  POINTS  OF  CONTACT.  Provide  a  list  of  points  of 
contact  as  illustrated  by  Figure  3-10. 

ANNEXES  OR  ATTACHMENTS.  Provide  as  appropriate. 


An  annex  is  written  specifically  for  the  TEMP,  whereas  an 
attachment  is  a  stand  alone  document. 
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APPENDIX  C 

PROGRAM  POINTS  OF  CONTACT 

(FORMAT) 

NAMF.  ORGANIZATION 

(Mailing  Address) 

PHONE 

(Commercial) 

(DSN) 

(FAX) 

(E-mail) 

Program  Manager 

PEO  Representative 

TIWG  Members 

User  Representative 

Figure  3-10.  Appendix  C.  Points  of  Contact  (Format) 


I 
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Chapter  4 

Format  and  Contents  for  Information  Mission  Area  (IMA) 
Programs 

Section  I 
Introduction . 

4-1.  General. 

a.  The  format  for  all  Amy  developed  Major  Automated 
Information  System  Review  Council  (MAISRC)  Test  and  Evaluation 
Master  Plans  (TEMPs)  will  be  in  accordance  with  Part  7,  DoD 
5000. 2-M. 


b.  DoD  5000. 2-M,  Part  7,  attachment  1,  is  replicated  in 
whole  in  the  following  sections,  except  as  noted  below.  Specific 
content  guidance  appropriate  for  Arny  preparation  of  MAISRC  TEMPs 
is  noted  in  the  boxed  remarks  following  a  given  paragraph. 

Figures  4-5  and  4-6  are  revised  showing  Arny  interpretation  of  the 
DOD  guidance. 

c.  signature  Page  formats  and  layout  for  programs  by  MAISRC 
decision  level  are  provided  at  figures  4-1  and  4-2.  Signature 
Page  format  and  layout  for  non-MAISRC  programs  is  provided  at 
figure  3-5.  Program  element  information  can  be  obtained  from  the 
current  year  version  of  AR  37-100-XX. 

d.  An  example  of  a  TIWG  Coordination  Sheet  is  at  figure  4-3. 
The  TIWG  Coordination  Sheet  should  show  the  specific  participants 
of  a  program,  for  example,  the  TIWG  Chair  should  show  the  PM, 
program  name,  the  functional  proponent  should  be  identified  ;  ISEC 
should  be  identified  as  the  Developmental  Evaluator,  OEC  as  the 
Operational  Evaluator,  etc.  Support  contractor  signatures  are  not 
acceptable.  Spell  out  the  name  and  organization  of  the  signatory 
(signature  block) . 

e.  A  TEMP  will  include  a  Signature  Page,  a  TIWG  Coordination 
Sheet  as  shown  in  figure  4-3,  and  an  Outline  as  shown  in  figure  4- 

4. 
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TEST  AND  EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (As  applicable) 

Program  Elements  REVISION  XX,  DATE  (As  applicable) 

Xxxxx 
Xxxxx 

★★★*★★*★★★**★*★***★***★*★*★★*★★★*★******♦***♦*******♦************ 

SUBMITTED  BY 


Program  Manager  DATE 


CONCURRENCE 


Program  Executive  Officer  DATE 
(or  Developing  Agency  if  no 
PEO) 


CDR  U.S.  Arny  Operational  DATE  Functional  Proponent  DATE 
Test  &  Evaluation  Command 

(OPTEC)  ■) 

APPROVED  BY 


Deputy  Under  Secretary  of  the  Amy  DATE 

(Operations  Research) 


FIGURE  4-1.  Signature  Page  format  for  Army  Major  Automated  Information  System 
Review  Committee  (MAISRC)  Programs 
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TEST  AND  EVALUATION  MASTER  PLAN 
FOR 

PROGRAM  TITLE 
SYSTEM  NAME 
DATE 

UPDATE  XX,  DATE  (As  applicable) 

Program  Elements  REVISION  XX,  DATE  (As  applicable) 

Xxxxx 
Xxxxx 

SUBMITTED  BY 


Program  Manager  DATE 


CONCURRENCE 


Program  Executive  Officer  DATE 

(or  Developing  Agency  if  no 
PEO) 


CDR  U.S.  Army  Operational  DATE  Functional  Proponent  DATE 

Test  &  Evaluation  Command 

(OPTEC) 


COMPONENT  APPROVAL 


Deputy  Under  Secretary  of  the  Army  DATE 

(Operations  Research) 

OSD  APPROVAL 


Director,  Operational  DATE  Director,  Test  and  Evaluation  Date 
Test  and  Evaluation  Under  Secretary  of  Defense 

(Acquisition) 


FIGURE  4-2.  signature  Page  Format  for  OSD  Major  Automated  Information  System 
Review  Committee  (MAISRC)  Programs 
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TIWG 

COORDINATION  SHEET 

TEMP  FOR 

PROGRAM  TITLE 

DATE 

UPDATE  XX,  DATE  (As  applicable) 
REVISION  XX,  DATE  (As  applicable) 

Signature 

Date 

Prooram  Manaaer 

CONCUR /NONCONCUR 

(TIWG  Chair) 

( Name / Or gani z a t i on ) 

Functional  Proponent/ 

CONCUR /NONCONCUR 

(Center /Agency) 

Developmental  Tester 

CONCUR /NONCONCUR 

(ISEC) 

Develop  Evaluator/Assessor 

CONCUR /NONCONCUR 

(ISEC) 

Operational  Tester 

CONCUR /NONCONCUR 

(TEXCOM) 

Operational  Evaluator 

CONCUR /NONCONCUR 

'  (OEC) 

Loaistician 

CONCUR /NONCONCUR 

Threat  Inteorator* 

CONCUR/NONCONCUR 

Other  ** 

CONCUR/NONCONCUR 

*  Required  for  Theater/Tactical  systems 

♦♦Include  participating  service  representatives  for  multiservice 
programs . 

FIGURE  4-3.  TEMP/TIWG  Coordination  Sheet. 
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TEST  AND  EVALUATION  MASTER  PLAN  OUTLINE  (FORMAT) 

Page 

Number 

PART  I  SYSTEM  INTRODUCTION  (2  pages  suggested  -  refer  to  annexes) 

a.  Mission  Description . xx 

b.  System  Threat  Assessment . xx 

c.  Minimum  Acceptable  Operational  Performance 

Requirements . xx 

d.  System  Description . xx 

e.  Critical  Technical  Parameters  (See  Figure  3-l0) . xx 

PART  II  INTEGRATED  TEST  PROGRAM  SUMMARY  (2  pages  suggested) 

a.  Integrated  Test  Program  Schedule  (see  Figure  3-11) . . .  .xx 

b .  Management . xx 

PART  III  DEVELOPMENTAL  TEST  AND  EVALUATION  OUTLINE  (10  pages 
suggested) 

a.  Developmental  Test  and  Evaluation  Overview . xx 

b.  Developmental  Test  and  Evaluation  to  Date . xx 

c.  Future  Developmental  Test  and  Evaluation . xx 

d.  Live  Fire  Test  and  Evaluation . xx 

PTUIT  IV  OPERATIONAL  TEST  AND  EVALUATION  OUTLINE  (10  pages 
suggested) 

a.  Operational  Test  and  Evaluation  Overview . xx 

b.  Critical  Operational  Issues . xx 

c.  Operational  Test  and  Evaluation  to  Date . xx 

d.  Future  Operational  Test  and  Evaluation . xx 

PART  V  TEST  AND  EVALUATION  RESOURCE  SUMMARY  (6  pages  suggested) 

a.  Test  Articles . . xx 

b.  Test  Sites  and  Instrumentation . . xx 

c.  Test  Support  Equipment . xx 

d.  Threat  Systems /Simulators . xx 

e.  Test  Targets  and  Expendables . xx 

f.  operational  Force  Test  Support . xx 

g.  Simulations,  Models  and  Testbeds . xx 

h.  Special  Requirements . xx 

i.  T&E  Funding  Requirements . xx 

j.  Manpower/Personnel  Training . xx 

APPENDIX  A  Bibliography . A-1 

APPENDIX  B  Acronyms . B-1 

APPENDIX  C  Points  of  Contact . C-1 

ANNEXES /ATTACHMENTS  (if  appropriate) 


FIGURE  4-4.  Test  and  Evaluation  Master  Plan  Outline. 
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Section  II 

Format  and  Contents  for  Information  Mission  Area  Systems 
4-2.  TEST  AND  EVALUATION  MASTER  PLAN  CONTENT  (FORMAT) 

PART  I  --  SYSTEM  INTRODUCTION 

Mission  Descript inn.  Reference  the  Mission  Need 
Statement  and  briefly  summarize  the  mission  need  described 
therein. 


Define  the  need  in  terms  of  mission,  objectives  and  general 
capabilities. 

System  capabilities  are  detailed  in  paragraph  2  and  4  of  the 
MNS  and  Part  1,  Section  4  of  the  System  Decision  Paper  (SDP) . 
Functional  process  improvement  is  detailed  in  Chapter  3  of  the 
MNS  or  Part  2,  Section  1  of  the  SDP. 


gvstem  Threat  Assessment.  Reference  the  system  threat 

assessment  and  briefly  summarize  the  threat  environment  described 
herein. 


Not  applicable  for  IMA  unless  system  developed  to  counter  a 
specific  threat.  If  a  STAR  is  prepared  for  the  system, 
summarize  the  operational  threat  environment  from  paragraph  4c 
of  the  STAR  and  the  system  specific  threat  from  paragraph  4e. 


c.  Minimum  Acceptable  Operational  Performance  Reouirementg . 

Reference  the  Operational  Requirements  Document  and  summarize 
the  critical  operational  effectiveness  and  suitability  parameters 
and  constraints  (manpower,  personnel,  training,  software,  computer 
resources,  transportation  (lift),  etc.)  described  therein. 


For  systems  conforming  to  DoD  STD  7935A,  operational 
requirements  are  specified  in  Section  2.2  of  the  Functional 
Description  (FD) . 

For  systems  conforming  to  DoD  STD  2167A,  operational 
requirements  are  specified  in  Sections  3.5.2  and  3.7-3.12  of  the 
Software  Requirements  Specification  (DI-MCCR-80025A) 

For  systems  using  accelerated  techniques  and  automated  tools, 
use  the  High  Level  Functional  Description  (HLFD) . 


i 
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d.  .c!vgtf>m  Dpsrrintion.  Briefly  describe  the  system  design. 

Include  the  following  items: 

(1)  Key  features  and  subsystems,  both  hardware  and  software 
(such  as  architecture,  interfaces,  security  levels,  reserves, 
etc.),  allowing  the  system  to  perform  its  required  operational 
mission. 

(2)  Interfaces  with  existing  or  planned  systems  that  are 
required  for  mission  accomplishment.  Address  relative  maturity 
and  integration  and  modification  requirements  for  nondevelopment 
items.  Include  interoperability  with  existing  and/or  planned 
systems  of  other  DoD  Components  or  allies. 

(3)  Critical  system  characteristics  (see  Section  4-C  of  DoD 
Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,")  or  unique  support  concepts  resulting  in  special  test 
and  analysis  requirements  (e.g.,  post  deployment  software  support, 
hardness  against  nuclear  effects;  resistance  to  countermeasures; 
development  of  new  threat  simulation,  simulators,  or  targets.) 


For  systems  conforming  to  DoD  STD  7935A,  )cey  features  of  the 
total  system  are  identified  in  Chapter  3  B  of  the  MNS  and 
Section  4  of  the  FD.  Interfaces  are  identified  in  Chapter  4  C 
of  the  MNS,  Section  5.4  of  the  FD,  and  Section  3  of  the  System 
Specification. 

For  systems  conforming  to  DoD  STD  2167A,  key  features  of  the 
total  system  are  identified  in  Chapter  3  B  of  the  MNS  and 
Section  3  of  the  System  Specification  (DI-CMAN-80008A) . 
Interfaces  are  identified  in  Section  3  of  the  Interface 
Requirements  Specification  (DI-MCCR-80026A) . 

For  systems  conforming  to  both  DoD  STD  7935A  .and  2167A, 
unique  system  characteristics  are  identified  in  Chapter  4  A  of 
the  MNS. 

Include  non  developmental  items  or  commercial-off-the-shelf 
software  and  any  required  interoperability  with  existing  or 
planned  systems  or  other  DOD  components  or  allies. 


e.  Critical  Technical  Parameters. 

(1)  List  in  a  matrix  format  (see  Figure  4-5)  the  critical 
technical  parameters  of  the  system  (including  software  maturity 
and  performance  measures)  that  have  been  evaluated  or  will  be 
evaluated  during  the  remaining  phases  of  developmental  testing. 
Critical  technical  parameters  are  derived  from  the  Operational 
Requirements  Document,  critical  system  characteristics  (see  Part  4 
of  DoD  Instruction  5000.2,  "Defense  Acquisition  Management 
Policies  and  Procedures")  and  technical  performance  measures  (see 
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Section  6-A  of  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures")  and  should  include  the 
parameters  in  the  acquisition  program  baseline  (see  Part  14,  DoD 
5000. 2-M).  Discuss  the  relationship  between  the  critical 
technical  parameters  and  the  minimum  acceptable  operational 
performance  requirements  in  the  Operational  Requirements  Document . 

(2)  Next  to  each  technical  parameter,  list  the  accompanying 
objectives  and  thresholds  as  illustrated  by  Figure  4-5. 


Critical 

Technical 

Parameters 

Total 

events 

Technical  objective 
for  each  test  event 

Location 

Schedule 

Decision 

supported 

Demonstrated 

value 

Measurable 
Parameter 
with  reference 

Single 
event  or 
test  phase 

Measurable 
Technical  value 

Test  facility 

Test  period 

Milestone, 
in-process 
review  or 
major  event 

(Include  the 
Actual  Value) 

Maximum 

Query 

response  time 
15  seconds 
(reference) 

Eur 

SDT 

SQT 

20  sec 

15  sec 

15  sec 

ABC  facility 
DEF  facility 
DEF  facility 

1QFY-XX 

2QFY-XX 

3QFY-XX 

MS  II 

IPR 

MS  lllc 

X 

Y 

Z 

Figure  4-5.  Sample  Critical  Technical  Parameters  Matrix 
(This  matrix  depicts  the  evaluation  criteria  to  assess  developmental  progress) 


Critical  Technical  Parameters  -  obtained  from  the  software 
specification  and  other  related  documents. 

For  systems  using  accelerated  techniques  and  automated  tools, 
critical  technical  parameters  are  derived  from  the  HLFD  and  its 
versions  as  it  transitions  to  become  the  Functional  Description 
(FD) . 

Reference  -  the  source  from  which  the  parameter  and  value  is 
derived. 

Total  events  -  the  developmental  tests  conducted  wherein  the 
parameters  are  tested.  Tests  should  be  outlined  in  Part  III. 

Technical  objective  for  each  event  -  the  value  expected  to  be 
attained  at  that  stage  of  development . 

Location  -  the  place  where  the  test  will  be  performed. 

Schedule  -  the  fiscal  quarter  when  the  test  will  be 
initiated. 

Decision  Supported  -  the  program  milestone  or  review  that 
will  consider  the  results  of  this  test. _ 
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Demonstrated  Value  -  state  the  actual  value  obtained  from 
testing. 

A  MS  I  (preliminary)  TEMP  is  not  expected  to  contain  detailed 
requirements.  The  TEMP  update  to  support  Milestone  II  should 
include  detailed  values. 


(3)  Highlight  critical  technical  parameters  that  must  be 
demonstrated  before  entering  the  next  acquisition  or  operational 
test  phase  and  ensure  that  the  actual  values  which  have  been 
demonstrated  to  date  are  included  in  the  last  coltimn. 


Critical  technical  parameters  are  defined  as  those  measurable 
critical  system  characteristics  including  software,  that  when 
achieved,  allow  the  attainment  of  the  minimum  acceptable 
operational  performance  requirements.  Software  critical 
technical  parameters  may  include  language,  architecture, 
interfaces,  supportability,  security  levels,  time,  memory,  and 
input /output  reserves. 

For  systems  conforming  to  DoD  STD  7935A,  a  matrix  relating 
the  critical  required  technical  parameters  may  be  derived  from 
information  found  in  the  System/Subsystem  Specification  and 
Chapter  2.5  of  the  User’s  Manual. 

For  systems  conforming  to  DoD  STD  2167A,  a  matrix  relating 
the  critical  required  technical  parameters  may  be  found  in 
Section  3.6  of  the  Software  Specification  (DI-MCCR-80025A) . 


PART  II  --  INTEGRATED  TEST  PROGRAM  SUMMARY 
a.  Integrated  Test  Program  Schedule 

(1)  As  illustrated  in  Figure  4-6,  display  on  a  chart 
the  integrated  time  sequencing  of  the  critical  test  and  evaluation 
phases  and  events,  related  activities,  and  planned  cumulative 
funding  expenditures  by  appropriation. 

(2)  Include  event  dates  such  as  milestone  decision 
points;  operational  assessments,  test  article  availability; 
software  version  releases;  appropriate  live  fire  test  and 
evaluation,  and  operational  test  and  evaluation;  low  rate  initial 
production  deliveries;  Full  Rate  Production  deliveries;  Initial 
Operational  Capability;  Pull  Operational  Capability;  and 
statutorily  required  reports. 

(3)  A  single  schedule  should  be  provided  for  multi- 
Service  of  Joint  and  Capstone  Test  and  Evaluation  Master  Plans 
showing  all  DoD  Component  system  event  dates. 
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Figure  4-6.  Integrated  Test  Program  Schedule  (Illustrative  Example) 


The  integrated  test  program  schedule  will  be  divided  into 
seven  major  areas:  Program  Milestones;  Program  Acquisition 
Events;  Contract  Release  and  Awards;  Program  Deliverables; 
Developmental  Test  and  Evaluation;  Operational  Test  and 
Evaluation  and  Program  Funding.  Figure  4-6  provides  an 
illustrated  example  of  an  integrated  test  program. 

Information/data  should  be  obtained  from  the  master  schedule. 
Section  F  of  the  Management  Plan  (MP) . 

Can  be  a  fold-out 

Must  cover  the  acquisition  and  test  &  evaluation  program 
through  full  operational  capability 

The  integrated  time  sequencing  of  critical  events  will  be 
reflected  as  appropriate  (for  example) : 

MILESTONES:  I,  II,  III,  Initial  Operational  Capability  (IOC) 
FORMAL  SOLICITATION  RELEASE: 
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Design 

Development 

Deployment 

CONTRACT  AWARD  AND  EVENTS: 

Design 

Development 

Deployment 

System  Software  Specification  (SSS) 
Software  Requirements  Review  (SRR) 
Software  Design  Review  (SDR) 

DELIVERIES: 

Prototype  (Designate  Quantity) 
Production  (Designate  Quantity) 

TECHNICAL  TEST  &  EVALUATION  (DT&E) : 


Pre-MS  III 

Software  Development  Test  (SDT) 

Software  Qualification  Test  (SQT) 

Software  Acceptance  Test  (SAT) 

Deployment /Operations  Phase 

Software  Change  Paclcage  (SCP) 

USER  TEST  &  EVALUATION  (OT&E) : 

Early  Operational  Assessment 
Operational  Assessment 

Pre-MS  II 

Early  User  Test  (EUT) 

Early  User  Experiment  (EUE) 

Pre-MS  III 

Limited  User  Test  (LUT) 

Initial  Operational  Test  (lOT) 

Deployment /Operations  Phase 

Follow-on  Operational  Test  (FOT) 

Lead  Site  Verification  Test  (LSVT) 

As  Required 

Supplemental  Site  Test 

FUNDING  -  Cumulative  by  fiscal  year 

O&S  -  include  all  O&S  expenditures,  not  just  T&E  related 

-  include  DT&E  and  OT&E  costs 
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-  include  Test  Articles  for  DT&E  and  lOT&E 

-  includes  OMA,  OMR  and  OMNG 

PROCUREMENT  -  as  described  in  the  program  Baseline  Agreement. 


b.  Management 


(1)  Discuss  the  test  and  evaluation  responsibility  of  all 
participating  organizations  (developers,  testers,  evaluators, 
users) . 


Identify  TIWG  members  and  their 

role.  Reference  the  TIWG 

charter  for  specific  responsibilities.  (See  AR  73-1  and  Part 

One,  Chapter  8)  The  TIWG  charter  must  be  included  as  a 

reference  in  Appendix  A,  Bibliography.  For  example: 

Program  Manager /TIWG  Chair 

PM  (System) 

Functional  Proponent 

Proponent  Agency 

Operational  Evaluator 

OEC 

Operational  Tester 

TEXCOM 

Developmental  Evaluator/Assessor 

ISEC 

Developmental  Tester 

ISEC 

Logistician/PDSS  Organization 

ISSC 

Threat  Integrator* 

TRADOC 

Participating  Service  OTA 

AFOTEC,  MCOTEA,  COMOPTEVFOR 

Participating  Service  Functional 

Proponent 

(if  multi -service) 

*  Required  for  Theater/Tactical 

systems . 

An  outline  of  T&E  responsibilities  of  all  participating  I 

organizations  is  defined  in  Section  2 

G  of  the  Program 

Manager/ Project  Manager  (PM)  charter. 

(2)  Provide  the  date  (fiscal  quarter)  when  the  decision  to 
proceed  beyond  low-rate  initial  production  is  planned.  (Low- rate 

initial  production  quantities  required  for  operational  test  must 
be  identified  for  the  Director  of  Operational  Test  and  Evaluation 
approval  prior  to  Milestone  II  for  acquisition  category  I  programs 
and  other  acquisition  category  programs  designated  for  Office  of 
the  Secretary  of  Defense  test  and  evaluation  oversight) . 


I 
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Provide  the  date  (fiscal  quarter)  when  the  decision  to 
proceed  to  Milestone  III  certification  is  planned.  If  the 
system  is  being  developed  through  an  incremental  acquisition 
strategy,  1)  provide  the  date  (fiscal  quarter)  when  the  decision 
to  proceed  to  MS  III  C.  is  planned  and  2)  briefly  outline  the 
extent  of  incremental  deployment  activities,  (prototype,  test 
bed  sites,  etc.)  prior  to  MS  III  C.  (Extent  of  incremental 
deployment  before  initial  Operational  Test  and  Evaluation  must 
be  identified  prior  to  MS  II  for  OSD  and  Army  MAISRC  systems) . 

The  quantity  of  items  needed  for  lOT  is  recommended  by  OPTEC 
in  coordination  with  the  program  manager  and  included  for 
approval  by  DOT&E  for  programs  having  OSD  test  and  evaluation 
oversight. 

The  quantity  of  items  needed  for  lOT  for  all  other  programs 
are  included  as  recommended  by  OPTEC. 


(3)  Identify  and  discuss  any  operational  issues  and 
vulnerability  and  lethality  Live  Fire  Test  requirements  that  will 
not  be  addressed  before  proceeding  beyond  low-rate  initial 
production. 


For  incremental  development  programs.  Milestone  II  is 
considered  as  equivalent  to  the  low  rate  initial  production 
(LRIP)  decision  point. 


PART  III  —  DEVELOPMENTAL  TEST  AND  EVALUATION  OUTLINE 

a.  Developmental  Test  and  Evaluation  Overview.  Explain  how 
developmental  test  and  evaluation  will:  verify  the  status  of 
engineering  and  manufacturing  development  progress;  verify  that 
design  risks  have  been  minimized;  and  substantiate  achievement  of 
contract  technical  performance  requirements;  and  be  used  to 
certify  readiness  for  dedicated  operational  test.  Specifically, 
identify: 

(1)  Any  technology /subsystem  that  has  not  demonstrated  its 
ability  to  contribute  to  system  performance  and  ultimately  fulfill 
mission  requirements. 

(2)  The  degree  to  which  system  hardware  and  software  design 
has  stabilized  so  as  to  reduce  manufacturing  and  production 
decision  uncertainties. 
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Summarize  the  entire  developmental  test  and  evaluation 
program. 

Present  a  narrative  walk-through  of  the  integrated  schedule, 
discussing  the  interrelationships  between  tests,  developmental 
and  operational,  and  between  tests  and  milestones.  Do  not 
duplicate  details  that  will  be  found  in  para  3c,  Future 
Developmental  Test  and  Evaluation.  The  purpose  of  the  Overview 
is  to  identify  how  the  individual  tests  fit  within  the  framework 
of  the  overall  program  and  the  continuous  evaluation  process. 
Some  of  the  topics  that  need  to  be  addressed  in  this  paragraph 
include : 

(1)  Early  developmental  tests  that  will  be  performed  to 
mitigate  technical  risks  in  the  program  that  are  defined  in  the 
Risk  Assessment  Annex  D.  Integrated  Program  Summary  (Reference 
DoD  5000. 2-M,  Part  4,  Annex  E.). 

(2)  Identification  of  developmental  tests  that  will  be  used 
to  demonstrate  that  the  test  item  is  safe,  that  the  technical 
manuals  are  verified  and  validated  and  ready  for  usage  in  a 
following  or  concurrent  Operational  Test. 

(3)  Identification  of  the  test,  usually  Software 
Qualification  Test  (SQT) ,  that  will  be  performed  to  validate 
that  the  system  meets  the  program's  technical  performance 
requirements  that  are  usually  contractually  mandated  in  a 
specification. 

The  following  areas  need  to  be  addressed  throughout 
Developmental  Test  and  Evaluation  (They  are  addressed  in  general 
in  the  DT&E  Overview  and  specifically  in  the  description, 
objective,  etc,  of  each  of  the  developmental  tests  addressed  in 
Future  DT&E.): 

Reliability,  Availability,  and  Maintainability  (Reference 
DoDI  5000.2,  Part  6,  Section  C.) 

Human  Factors  (Reference  DoDi  5000.2,  Part  6,  Section  H) 

System  Safety,  Health  Hazards  and  Environment  (Reference  DoDi 
5000.2,  Part  6,  Section  I) 

Discuss  the  metrics  that  will  be  used  to  determine  software 
status  and  evaluate  progress  toward  software  maturity  in  support 
of  key  decision  points.  Show  how  the  metrics  in  each  phase 
relate  to  those  in  previous  and  subsequent  phases. 


b.  Developmental  Test  and  Evaluation  to  DatP.  Identify 
completed  developmental  test  and  evaluation  by  noting  on  the 
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matrix  of  critical  technical  parameters  those  parameters  that  have 
been  demonstrated. 


Update  the  Critical  Technical  Parameters  matrix  in  Part  I. 

For  parameters  not  met,  provide  a  brief  explanation  as  to  why 
and  performance  impact.  Identify  future  test  that  will  re¬ 
address  parameters. 

If  during  any  prior  T&E  phase  or  event,  mission  critical 
deficiencies  were  identified,  a  discussion  of  the  nature  of  each 
deficiency,  corrective  action  required,  or  the  schedule  for  the 
DT&E  retest  verification,  should  be  included  as  derived  from 
Section  3  of  the  Test  Analysis  report.  {DI-MCCR-80017A  for  DoD 
STD  2 167 A) 

A  detailed  discussion  of  the  results  of  testing  is  not 
required. 

Test  and  evaluation  reports  prepared  to  date  must  be  included 
as  references  in  Appendix  A.  Bibliography. 


c.  Future  Developmental  Test  and  Evaluation.  Discuss  all 
remaining  developmental  test  and  evaluation  that  is  plained, 
beginning  with  the  date  of  the  current  Test  and  Evaluation  Master 
Plan  revision  and  extending  through  completion  of  production. 
Place  emphasis  on  the  next  phase  of  testing.  For  each  phase, 
include; 


For  each  test  within  each  remaining  aco^isition  phase  address 
the  following  items;  configuration  description,  DT.&E 
objectives,  DT&E  events,  scope,  basic  scenarios  and  limitations 
e.g.,  (1)  Development 

(a)  Software  Development  Test  (SDT) 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing,  and  Basic  Scenarios 

(4)  Limitations 

(b)  Software  Qualification  Test  (SQT) 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testings  and  Basic  Scenarios 

(4)  Limitations 

For  those  critical  technical  parameters  where  demonstrated 
value  did  not  meet  the  threshold  or  objective,  planned  testing 
must  insure  that  these  parameters  will  be  re-addressed. 
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(1)  Configuration  Description.  Summarize  the  functional 
capabilities  of  the  system's  developmental  configuration  and  how 
they  differ  from  the  production  model. 

List  the  difference  between  the  system  to  be  tested  and  the 
objective  system,  to  include  software. 

For  systems  conforming  to  DoD  STD  7935A  and  2167A,  a  summary 
of  future  DT&E  system  hardware  and  software  functional 
capability  and  how  it  is  expected  to  differ  from  the 
configuration  planned  for  deployment  may  be  found  in  the  Test 
Plan. _ 


(2)  Eey.elQPmental  Test  and  Evaluation  Qb-iectives.  State  the 
test  objectives  for  this  phase  in  terms  of  the  critical  technical 
parameters  to  be  confirmed.  Identify  any  specific  technical 
parameters  which  the  milestone  decision  authority  has  designated 
as  exit  criteria  and/or  directed  to  be  demonstrated  in  a  given 
phase  of  testing. 


Discuss  problem  areas,  if  any,  identified  by  the  use  of 
software  metrics.  Describe  how  future  developmental  test  and 
evaluation  events  will  measure  progress  toward  elimination  of 
these  problem  areas. 


(3)  Developmental  Test  and  Evaluation  Events.  Scope  of 
Testing .  and  Basic  Scenarios.  Summarize  the  test  events,  test 
scenarios  and  the  test  design  concept  Quantify  the  testing  (i.e., 
number  of  test  hours,  test  events,  test  firings).  List  the 
specific  threat  systems,  surrogates,  countermeasures,  component  or 
subsystem  testing,  and  testbeds  the  use  of  which  are  critical  to 
determine  whether  developmental  test  objectives  are  achieved.  As 
appropriate,  particularly  if  an  agency  separate  from  the  test 
agency  will  be  doing  a  significant  part  of  the  evaluation, 
described  the  methods  of  evaluation.  List  all  models  and 
simulations  to  be  used  and  explain  the  rationale  for  their 
credible  use.  Describe  how  performance  in  natural  environmental 
conditions  representative  of  the  intended  area  of  operations 
(e.g.,  temperature,  pressure,  humidity,  fog,  precipitation, 
clouds,  blowing  dust  and  sand,  icing,  wind  conditions,  steep 
terrain,  wet  soil  conditions,  high  sea  state,  storm  surge  and 
tides,  etc.)  and  interoperability  and  compatibility  with  other 
weapon  and  support  systems  as  applicable  will  be  tested. 


The  resources  identified  must  correspond  to  those  listed  in 
Part  V. 

Include  discussion  of  any  test  databases  and/or  remote 
terminal  emulators  to  be  used  and  their  relationship  to  the 
objective  system  environment. _ 
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(4)  T. imitations.  Discuss  the  test  limitations  that  may 
significantly  affect  the  evaluator's  ability  to  draw  conclusions, 
the  impact  of  these  limitations,  and  resolution  approaches. 

d.  Live  Fire  Test  and  Evaluation.  Include  a  description  of 
the  overall  live  fire  test  and  evaluation  strategy  for  the  item; 
critical  live  fire  test  and  evaluation  issues;  reguired  levels  of 
system  vulnerability/lethality;  the  management  of  the  live  fire 
test  and  evaluation  program  live  fire  test  and  evaluation 
schedule,  funding  plans  and  requirements;  related  prior  and  future 
live  fire  test  and  evaluation  efforts;  the  evaluation  plan  and 
shot  selection  process;  and  major  test  limitations  for  the  conduct 
of  live  fire  test  and  evaluation.  Live  fire  test  and  evaluation 
resource  requirements  (including  test  articles  and 
instrumentation)  will  be  appropriately  identified  in  the  Test  and 
Evaluation  Resource  Summary. 


Generally  not  applicable  for  IMA  systems,  except  when 
development  includes  protective  shelters. 


Part  IV  --  OPERATIONAL  TEST  AND  EVALUATION  OUTLINE 
a.  Operational  Test  and  Evaluation  Overview 

(1)  The  primary  purpose  of  operational  testing  and 
evaluation  is  to  verify  that  operationally  effective  and 
operationally  suitable  systems  are  approved  for  production  that 
meet  the  mission  needs  and  minimum  operational  performance 
requirements  of  the  operating  forces. 

(2)  The  Test  and  Evaluation  Master  Plan  will  show  how 
program  schedule,  test  management  structure,  and  required 
resources  are  related  to  operational  requirements,  critical 
operational  issues,  test  objectives,  and  milestone  decision 
points.  Testing  will  evaluate  the  system  (operated  by  typical 
users)  in  an  environment  as  operationally  realistic  as  possible, 
including  threat  representative  hostile  forces  and  the  expected 
range  of  natural  environmental  conditions. 


Summarize  the  entire  operational  test  and  evaluation  program. 
Present  a  narrative  walk-through  of  the  integrated  schedule 
discussing  the  interrelationships  between  contractor, 
government,  developmental  and  operational  tests,  models  and 
simulations  and  the  milestones  they  support.  Do  not  duplicate 
the  details  that  are  provided  in  paragraph  4.d,  Future 
Operational  Test  and  Evaluation.  The  purpose  of  the  overview  is 
to  give  a  quick,  concise  look  at  the  overall  test  program, 
explaining  the  many  interrelationships  and  opportunities  to 
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conduct  continuous  evaluation  (CE) .  Some  of  the  topics  that 
need  to  be  addressed  include: 

(1)  Identification  of  contractor  and  developmental  tests 
that  will  be  used  as  part  of  an  operational  evaluation  or 
assessment . 

(2)  Identification  of  simulations  that  will  be  used  to 
augment  and  extend  operational  testing  as  part  of  an  operational 
evaluation  or  assessment. 


b.  Critical  Operational  Issues 

(1)  List  in  this  section  the  critical  operational  issues. 
Critical  operational  issues  are  the  operational  effectiveness  and 
operational  suitability  issues  (not  parameters,  objectives  or 
thresholds)  that  must  be  examined  in  operational  test  and 
evaluation  to  evaluate/assess  the  system's  capability  to  perform 
its  mission. 

(2)  A  critical  operational  issue  is  typically  phrased  as  a 
question  that  must  be  answered  in  order  to  properly  evaluate 
operational  effectiveness  (e.g.,"Will  the  system  detect  the  threat 
in  a  combat  environment  at  adequate  range  to  allow  successful 
engagement?*)  and  operational  suitability  (e.g.,  "Will  the  system 
be  safe  to  operate  in  a  combat  environment?"). 

(3)  Some  critical  operational  issues  will  have  critical 
technical  parameters  and  minimum  acceptable  operational 
performance  requirements  or  thresholds.  Individual  attainment  of 
these  attributes  does  not  guarantee  that  the  critical  operational 
issue  will  be  favorably  resolved.  The  judgment  of  the  operational 
test  agency  is  used  by  the  DoD  Component  to  determine  if  the 
critical  operational  issue  is  favorably  resolved. 

(4)  If  every  critical  operational  issue  is  resolved 
favorably,  the  system  should  be  operationally  effective  and 
operationally  suitable  when  employed  in  its  intended  environment 
by  typical  users. 


Functional  Proponent  (FP)  developed  and  approved  Critical 
Operational  Issues  and  Criteria  (COIC)  are  required  for  all  Army 
and  OSD  MAISRC  programs  for  MS  I. 

DISC 4  approved  COIC  are  required  for  Arity  and  OSD  MAISRC 
systems  at  MS  II  and  beyond. 

Include  the  approved  COICs  in  their  entirety  in  the  TEMP; 
this  includes  Issue,  Scope,  Criteria  and  Rationale. 

i 
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Reference  the  COIC  approval  document  in  Appendix  A, 
Bibliography. 


c.  Operational  Test  and  Evaluation  to  Date.  Identify  and 
date  test  reports  that  detail  the  results  of  testing  and 
operational  assessments  to  date.  Indicate  critical  operational 
issues  that  were  resolved  (satisfactory,  unsatisfactory,  yes,  no 
etc.),  partially  resolved,  or  unresolved  at  the  completion  of  each 
phase  of  testing. 


The  results  related  to  the  resolution  of  the  criteria  in 
addition  to  the  overall  issue  should  be  discussed. 

Ensure  that  all  test  reports  referenced  are  listed  in 
Appendix  A,  Bibliography.  Reports  must  be  available  if 
requested. 

For  software,  within  the  context  of  previously  identified 
operational  issues,  summarize  what  has  been  learned  from  the 
operational  test  accomplished  to  date  about  the  maturity  of  the 
software.  Show  how  operational  test  results  from  interim 
hardware  and  software  configurations  apply  to  configurations 
intended  for  deployment.  Identify  differences  between  tested 
software,  software  planned  for  the  current  phase,  and  software 
to  be  deployed.  Discuss  the  importance  of  these  differences. 


d.  Future  Operational  Test  and  Evaluation.  For  each 
remaining  phase  of  operational  test  and  evaluation,  separately 
address  the  following: 


Identify  operational  tests  that  will  be  conducted  and  the 
developmental  tests  that  will  be  used  as  data  for  operational 
evaluation  or  assessment,  when  developmental  tests  are 
identified,  subparagraph  (3)  Events,  Scope  of  Testing  and 
Scenarios  should  define  the  data  that  will  be  taJcen  from  the 
developmental  test  for  the  evaluation  or  assessment.  This  will 
ensure  that  the  developmental  testers  and  evaluators,  by  their 
signature  on  the  TEMP,  have  agreed  to  collect  and  provide  that 
data  to  the  operational  evaluator. 

Describe  how  models  will  be  accredited  for  use  in  this 
specific  application.  The  approval  vehicle  for  accreditation  is 
an  Accreditation  Plan  as  outlined  in  DUSA(OR)  memo  dated  30  Oct 
89,  subject:  Verification,  Validation,  and  Accreditation  of 
Models.  Reference  the  Accreditation  Plan  in  Appendix  A, 
Bibliography. 

Part  V,  Resource  Summaiy  will  identify  the  resources 
necessary  to  perform  the  validation  and/or  accreditation. 
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If  more  than  one  test  is  in  a  phase,  the  following  layout 
should  be  included  for  each  test.  For  example: 

(1)  Development 

(a)  Limited  User  Test  (LUT) 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing  and  Scenarios 

(4)  Limitations 

(Note;  Either  list  each  sub-element  for  the  developmental  test  to 
be  used  for  data  or  refer  to  the  applicable  paragraph  in  Part 
III  that  contains  the  information.) 

(b)  Initial  Operational  Test  and  Evaluation 

(1)  Configuration  description  (of  test  item) 

(2)  Test  and  Evaluation  Objectives 

(3)  Events,  Scope  of  Testing  and  Scenarios 

(4)  Limitations. 


(1)  Configuration  Description.  Identify  the  system  to  be 
tested  during  each  phase,  and  describe  any  differences  between  the 
tested  system  and  the  system  that  will  be  fielded  including,  where 
applicable,  software  maturity  performance  and  criticality  to 
mission  performance,  and  the  extent  of  integration  with  other 
systems  with  which  it  must  be  interoperable  or  compatible. 
Characterize  the  system  (e.g.,  prototype,  engineering  development 
model,  production  representative  or  production  configuration) 

(2)  Operational  Test  and  Evaluation  Objectives.  State  the 
test  objectives  including  the  minimum  acceptable  operational 
performance  requirements  and  critical  operational  issues  to  be 
addressed  by  each  phase  of  operational  test  and  evaluation  and  the 
milestone  decision  review (s)  supported.  Operational  test  and 
evaluation  that  supports  the  beyond  low  rate  initial  production 
decision  should  have  test  objectives  that  examine  all  areas  of 
operational  effectiveness  and  suitability. 


Human  performance  issues  must  be  addressed  (Reference  DoDI 
5000.2,  Part  7,  Section  B) . 

Discuss  the  relationship  between  OT&E  objectives  and  the 
software  characteristics  which  affect  critical  operational 
issues. 

For  FOT&E,  identify  major  deficiency  corrections  to  be 
verified.  UTs  should  be  designed  to  assure  that  software  is 
fault  tolerant  and  supportable. 
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(3)  Operational  Test  and  Evaluation  Events.  Scope  of 
Testing,  and  Scenarios.  Summarize  the  scenarios  and  identify  the 
events  to  be  conducted,  type  of  resources  to  be  used,  the  threat 
simulators  and  the  simulation (s)  to  be  employed,  the  type  of 
representative  personnel  who  will  operate  and  maintain  the  system, 
the  status  of  the  logistic  support,  the  operational  and 
maintenance  documentation  that  will  be  used,  the  environment  under 
which  the  system  is  to  be  employed  and  supported  during  testing, 
the  plans  for  interoperability  and  compatibility  testing  with 
other  United  States/Allied  weapon  and  support  systems  as 
applicable,  etc.  Identify  planned  sources  of  information  (e.g., 
developmental  testing,  testing  of)  related  systems,  modeling, 
simulation,  etc.)  that  may  be  used  by  the  operational  test  agency 
to  supplement  this  phase  of  operational  test  and  evaluation. 
Whenever  models  and  simulations  are  to  be  used,  explain  the 
rationale  for  their  credible  use.  If  operational  test  and 
evaluation  cannot  be  conducted  or  completed  in  this  phase  of 
testing  and  the  outcome  will  be  an  operational  assessment  instead 
of  an  evaluation,  this  should  clearly  be  stated  and  the  reason (s) 
explained. 


Include  a  description  of  the  relationship  between  software 
functions  being  tested  and  test  scenario  events  that  will  cause 
that  function  to  be  exercised.  Identify  load  levels  to  be  used 
and  their  relationship  to  the  required  operational  environment. 


(4)  Limitations.  Discuss  the  test  limitations  including 
threat  realism,  resource  availability,  limited  operational 
(military,  climatic,  nuclear,  etc.)  environments,  limited  support 
environment,  maturity  of  tested  system,  safety,  etc.,  that  may 
impact  the  resolution  of  affected  critical  operational  issues. 
Indicate  the  impact  of  the  test  limitations  on  the  ability  to 
resolve  critical  operational  issues  and  the  ability  to  formulate 
conclusions  regarding  operational  effectiveness  and  operational 
suitability.  Indicate  the  critical  operational  issues  affected  in 
parenthesis  after  each  limitation. 


Identify  any  factors  which  may  inhibit  realistic  OT  of  the 
software.  Constraints  imposed  by  software  maturity  or 
availability  of  resources  and  simulators  should  be  given  along 
with  their  impact  on  critical  operational  issues. 


PART  V  —  TEST  AND  EVALUATION  RESOURCE  SUMMARY 

Provide  a  summary  (preferably  in  a  table  or  matrix  format)  of 
all  key  test  and  evaluation  resources,  both  government  and 
contractor,  which  will  be  used  during  the  course  of  the 
acquisition  program.  Specifically,  identify  the  following  test 
resources : 
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A  matrix  or  table  is  preferred  to  identify  planned  use  of 
available  key  assets  and  requirements  for  a  new  or  unique 
capability  or  item  that  needs  to  be  acquired  or  developed  to 
support  the  test  program.  The  following  should  be  addressed: 

test  articles 

test  sites  and  instrumentation 

test  support  equipment 

simulations,  models  and  testbeds 

Developmental  tester  and  operational  tester  should  provide 
input  specific  to  their  requirements.  Indicate  which 
requirements  were  identified  by  each  tester. 

Existing  capabilities  that  are  key  to  accomplishing  the  test 
program  need  be  included  and  specifically  all  those  where  use  is 
known  to  be  restricted  or  significant  upgrade  or  improvement  is 
needed. 

The  matrix  should,  at  a  minimum,  identify  the  item,  the 
quantity  or  nximber  required,  the  location,  the  test  event  or 
time  frame  when  needed,  the  resources  required  to  be  obtained 
and  the  organization/activity  responsible  for  acquisition  or 
development . 

Resource  requirements  are  found  in  the  Management  Plan  (MP) . 


a.  Test  Articles.  Identify  the  actual  number  of  and  timing 
requirements  for  all  test  articles,  including  key  support 
equipment  and  technical  information  required  for  testing  in  each 
phase  by  major  type  of  developmental  test  and  evaluation  and 
operational  test  and  evaluation.  If  key  subsystems  (components, 
assemblies,  subassemblies  or  software  modules)  are  to  be  tested 
individually,  before  being  tested  in  the  final  system 
configuration,  identify  each  subsystem  in  the  Test  and  Evaluation 
Master  Plan  and  the  quantity  required.  Specifically  identify  when 
prototype,  engineering  development,  pre-production,  or  production 
models  will  be  used. 

b.  Test  Sites  and  Instrumentation.  Identify  the  specific 
test  ranges/ facilities  to  be  used  for  each  type  of  testing. 

Compare  the  requirements  for  test  ranges/ facilities  dictated  by 
the  scope  and  content  of  planned  testing  with  existing  and 
programmed  test  range/ facility  capability,  and  highlight  any  major 
shortfalls,  such  as  inability  to  test  under  representative  natural 
environmental  conditions. 

Identify  instrumentation  that  must  be  acquired  specifically 
to  conduct  the  planned  test  program. 
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This  includes  software  facilities  and  tools  to  support 
testing  identified  in  Parts  III  and  IV. 

Address  instrumentation  that  must  be  developed  and/or 
procured.  Clearly  identify  the  test  investment  requirement. 


c.  Test  Support  Equipment.  Identify  test  support  equipment 
that  must  be  acquired  specifically  to  conduct  the  test  program. 


Only  address  new  test  support  equipment.  This  includes 
software  test  drivers,  emulators,  stimulators  or  diagnostics,  if 
applicable,  to  support  identified  testing.  Identify  unique  or 
special  calibration  requirements  associated  with  this  test 
support  equipment. 


d.  Threat  Svstems/Simulators .  Identify  the  type,  number, 
availability,  and  fidelity  requirements  for  all  threat 
systems/simulators.  Compare  the  requirements  for  threat 
systems/simulators  with  available  and  projected  assets  and  their 
capabilities.  Highlight  any  major  shortfalls.  Each  threat 
simulator  shall  be  subjected  to  validation  procedures  to  establish 
and  docximent  a  baseline  comparison  with  its  associated  threat  and 
to  ascertain  the  extent  of  the  operational  and  technical 
performance  differences  between  the  two  throughout  the  simulator's 
life-cycle. 


Generally  not  applicable  for  IMA  systems,  except  for 
Theater /Tactical  systems. 


e.  Test  Targets  and  Expendables.  Identify  the.  type,  number, 
and  availability  requirements  for  all  targets,  flares,  chaff, 
sonobuoys,  smoke  generators,  acoustic  countermeasures,  etc.,  that 
will  be  required  for  each  phase  of  testing.  Identify  any  major 
shortfalls. 


Not  applicable  for  IMA  systems. 


f.  Operational  Force  Test  Support.  For  each  test  and 
evaluation  phase,  identify  the  type  and  timing  of  aircraft  flying 
hours,  ship  steaming  days,  and  on-orbit  satellite 
contacts/coverage,  and  other  critical  operating  force  support 
required . 


Include  size,  location  and  type  of  unit  required. 
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g.  simulation.  Models,  and  Testbeds.  For  each  test  and 
evaluation  phase,  identify  the  system  simulations  required, 
including  computer-driven  simulation  models  and  har dware/ software - 
in-the-loop  testbeds.  Identify  the  resources  required  to  validate 
and  certify  their  credible  usage  or  application  before  their  use. 


Include  only  those  simulations,  models  and  testbeds  that  will 
be  used  to  extend  testing  and/or  used  in  evaluation.  This 
includes  feeder  models. 


h.  Special  Requirements.  Discuss  requirements  for  any 

significant  non- instrumentation  capabilities  and  resources  such 
as:  special  data  processing/data  bases,  unique 

mapping/charting/geodesy  products,  extreme  physical  environmental 
conditions  or  restricted/special  use  air/sea/landscapes. 

i.  Test  and  Evaluation  Funding  Requirements.  Estimate,  by 
fiscal  year  and  appropriation  line  nxamber  (program  element) ,  the 
funding  required  to  pay  direct  costs  of  planned  testing.  State, 
by  fiscal  year,  the  funding  currently  appearing  in  those  lines 
(program  elements).  Identify  any  major  shortfalls. 


Use  of  a  table  or  matrix  is  preferred. 
Show  potential  shortfalls. 


j.  Manpower /Personnel  Training.  Identify  manpower/personnel 
and  training  requirements  and  limitations  that  affect  test  and 
evaluation  execution. 

The  preliminary  Test  and  Evaluation  Master  Plan  should 
project  the  Icey  resources  necessary  to  accomplish  demonstration 
and  validation  testing  and  early  operational  assessment.  The 
preliminary  Test  and  Evaluation  Master  Plan  should  estimate,  to 
the  degree  known  at  Milestone  I,  the  key  resources  necessary  to 
accomplish  developmental  test  and  evaluation,  live  fire  test  and 
evaluation,  and  operational  test  and  evaluation.  These  should 
include  elements  of  the  National  Test  Facilities  Base  (which 
incorporates  the  Major  Range  and  Test  Facility  Base  (MRTFB) , 
capabilities  designated  by  industry  and  academia,  and  Major  Range 
and  Test  Facility  Base  test  equipment  and  facilities),  unique 
instrumentation,  threat  simulators,  and  targets.  As  system 
acquisition  progresses,  the  preliminary  test  resource  requirements 
shall  be  reassessed  and  refined  and  subsequent  Test  and  Evaluation 
Master  Plan  updates  shall  reflect  any  changed  system  concepts, 
resource  requirements,  or  updated  threat  assessments.  Any 
resource  shortfalls  which  introduce  significant  test  limitations 
should  be  discussed  with  planned  corrective  action  outlined. 
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This  paragraph  contains  overall  guidance  for  preparing  a 
preliminary  TEMP,  i.e.,  a  TEMP  to  support  Milestone  I.  It  is 
not  a  separate  paragraph  to  be  addressed  in  the  TEMP. 


APPENDIX  A  —  BIBLIOGRAPHY 

a.  Cite  in  this  section  all  documents  referred  to  in  the 
Test  and  Evaluation  Master  Plan. 

b.  Cite  all  reports  documenting  developmental  and 
operational  testing  and  evaluation. 

APPENDIX  B  —  ACRONYMS.  List  and  define  all  acronyms  used  in 
the  Test  and  Evaluation  Master  Plan. 

APPENDIX  C  —  POINTS  OF  CONTACT.  Provide  a  list  of  points  of 
contact  as  illustrated  by  Figure  4-7. 

ANNEXES  OR  ATTACHMENTS.  Provide  as  appropriate. 


An  annex  is  written  specifically  for  the  TEMP,  whereas  an 
attachment  is  a  stand  alone  document. 
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PROGRAM  POINTS  OF  CONTACT  (FORMAT) 


NAME  ORGANIZATION 

(Mailing  address)  (Commercial) 

(DSN) 

(FAX) 

(E-mail) 

Program  Manager 
PEO  Representative 
TIWG  Members 
User  Representative 


Figure  4-7.  Appendix  C.  Points  of  Contact  (Format) 
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Chapter  5 
TEMP  Checklist 

The  checklist  is  intended  as  a  guide  to  both  TEMP  developers  and 
TEMP  reviewers.  The  checklist  when  properly  used  should  ensure 
that  all  necessary  and  appropriate  requirements  in  the  approved 
test  and  evaluation  strategy  are  adequately  considered  and 
efficiently  addressed  in  test  and  evaluation  planning  and  program 
execution. 
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PROGRAM _ 

CURRENT  TEMP  MILESTONE _  DATE  OF  REVIEW. 

REVIEWED  BY _ 


Yes  No  N/A 

Signature  Page 

a.  Does  the  page  contain  the  necessary  _  _  _ 

signatures  for  the  acquisition  category  of  the 
program? 


b.  Is  a  date  at  the  top  of  the  page?  _ 

c.  Is  there  a  update  number,  if  this  is  _ 

not  an  initial  submission? 

d.  Is  there  a  revision  number  if  this  _ 

version  contains  changes  based  comments  from  HQDA 
and/or  OSD  on  reviews? 

TIWG  COORDINATION  SHEET 

Are  there  signature  blocks  for: 

Program  Manager  _ 

Combat  Developer  _ 

Developmental  Tester  _ 

Tech  Evaluator/Assessor  _ 

Operational  Tester  _ 

Operational  Evaluator  _ 

Logistician  - 

Threat  Integrator  - 

Others  as  required  _ 

Part  I.  System  Introduction 

a.  Mission  Description. 

(1)  Mission  of  the  deployed  system  briefly  _ 

described? 

(2)  Does  the  mission  description  agree  with  _ 

the  Mission  Need  Statement  (MNS)  and/or 
Operational  Requirements  Document  (ORD) ? 

(3)  Is  the  need  defined  in  terms  of  _ 

mission,  objectives,  and  general  capabilities? 
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(4)  Is  the  Mission  Need  Statement  (MNS)  _ 

referenced  in  the  Appendix  A  -  Bibliography? 

b.  System  Threat  Assessment. 

(1)  Is  the  system  threat  briefly  described?  _ 

(2)  Is  the  operational  threat  environment  _ 

summarized  from  the  STAR? 

(3)  Is  the  threat  at  IOC,  follow-on  at  IOC  _ 

plus  10  and  the  reactive  threat  listed? 

(4)  Is  the  STAR  referenced  in  Appendix  A  -  _ 

Bibliography? 

c.  Minimum  Acceptable  Operational  Performance 
Requirements. 

(1)  Are  the  critical  operational  effective-  _ 

ness  and  suitability  parameters  and  constraints 
summarized  from  the  ORD? 

(2)  Is  the  ORD  referenced  in  Appendix  A  -  _ 

Bibliography? 

d.  System  Description. 

(1)  System  design  briefly  described?  _ 

(2)  Key  features  both  hardware  and  software  _ 

and  subsystems  allowing  the  system  accomplishment 

of  operational  mission  described? 

(3)  Interfaces  with  existing  or  planned  _ 

systems  that  are  required  for  mission 
accomplishments  described? 

(4)  Are  critical  characteristics  of  the  _ 

system  or  unique  support  concepts  resulting  in 
special  test  and  evaluation  requirements  listed? 

(5)  System  software,  if  used,  described?  _ 

(6)  Are  existing  and/or  planned  systems  of  _ 

other  DoD  Components  or  allies  for  which  inter¬ 
operability  with  this  end  item  is  required  listed? 

(7)  Has  the  description  of  the  overall  _ 

system  included  Mission  Critical  Computer  Resources 
(MCCR)  for  software  utilized  by  the  system? 

(8)  Have  key  processors,  software  _ 

(including  firmware)  configuration  items,  system 
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interfaces,  internal  and  external  message  standards 
and  protocols  been  identified? 

e.  Critical  Technical  Parameters. 

(1)  Critical  technical  parameters  that  _ 

have  been/will  be  evaluated  during  all  phases  of 
development  listed  in  the  matrix? 

(2)  Accompanying  technical  threshold  listed  _ 
next  to  each  technical  parameter? 

(3)  Are  results  from  developmental  test 
addressing  a  given  parameter  posted? 

Part  II.  Integrated  Test  Program  Summary 

a.  Integrated  Test  Program. 

(1)  Is  an  integrated  test  program 
presented  for  the  seven  major  areas  of  interest? 

MILESTONES 
ACQUISITION  EVENTS 
CONTRACT  AWARDS  AND  EVENTS 
DELIVERIES 

TECHNICAL  TEST  AND  EVALUATION 
LIVE  FIRE  TEST  AND  EVALUATION 
USER  TEST  AND  EVALUATION 

(2)  Does  the  funding  data  correspond  to 
progrcimmatic  forecasts  and  contain  all  categories 
of  funding  as  described  in  chapter  8,  AR  73 -XX? 

(a)  MRTFB  Reimbursable  identified? 

(b)  RDTE  identified? 

(c)  Procurement  identified? 

b.  Management. 

(1)  T&E  responsibilities  of  all  partici¬ 
pating  organizations  outlined? 

(2)  Is  the  TIWG  charter  referenced 
in  Appendix  A  -  Bibliography? 

(3)  Is  a  clear  definition  of  LRIP  and 
full-rate  production  provided? 

(4)  Is  the  date  when  the  decision  to 
proceed  beyond  LRIP  provided? 

(5)  Have  participating  organizations 
responsible  for  software  T&E  been  identified? 


5-4 


DA  Pamphlet  73-1,  Part  Two,  16  Oct  92 


(DRAFT) 


(7)  Vulnerability  and  lethality  Live  Fire  _ 

Test  requirements  and  operational  issues  that 
cannot  be  addressed  before  proceeding  beyond 

LRIP  explanation  provided? 

(8)  Are  responsibilities  for  configuration  _ 

management  of  test  articles  designated? 

(9)  Are  responsibilities  for  establishing  _ 

HUC  designated? 

(10)  Is  the  HUC  determination  that  further  _ 

review  not  required  documented  here  and  document 
listed  in  Appendix  A  -  Bibliography? 

(11)  Do  the  quantities  required  for  DT&E  _ 

and  lOT&E  correspond  to  those  quantities  designated 
in  PART  V? 


Part  III.  Developmental  Test  and  Evaluation  Outline 
a.  Developmental  Test  and  Evaluation  (DT&E)  Overview. 

(1)  Explanation  included  of  how  planned 
DT&E  will  verify: 

Status  of  engineering  design  and  _ 

development 

Design  rislcs  have  been  minimized  _ 

Achievement  of  technical  performance  _ 

Readiness  for  lOT  _ 

(2)  Are  technologies  identified  which  have  _ 

not  been  demonstrated? 

(3)  Is  the  degree  to  which  the  system  has  _ 

stabilized  been  addressed? 

(4)  Has  a  discussion  of  the  indicators  that  _ 

will  be  used  to  determine  software  status  and 
evaluate  progress  toward  software  maturity  in 
support  of  key  decision  points  been  identified? 

(5)  Is  a  narrative  walkthrough  of  the  _ 

integrated  schedule  discussing  the  interrelation¬ 
ships  between  tests,  developmental,  and  operational, 
and  between  tests  and  milestones  presented? 

(6)  Are  early  developmental  tests  scheduled  _ 
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which  will  mitigate  the  technical  risks  identified 
in  the  Integrated  Program  Summary,  Annex  D? 

(7)  Is  the  Integrated  Program  Summary  _ 

referenced  in  Appendix  A  -  Bibliography? 

(8)  Are  developmental  tests,  that  feed  into  _ 

operational  tests  or  evaluations,  identified? 

(9)  Are  tests,  that  validate  supportability  _ 

requirements  (i.e.,  TMs  and  support  packages) 
identified? 

(10)  Is  the  test,  that  will  validate  the  _ 

program's  requirements  against  the  system 
specification,  identified? 

(11)  Has  survivability/lethality  testing  _ 

been  highlighted? 

b.  Developmental  Test  and  Evaluation  to  Date. 

(1)  Are  the  demonstrated  technical  _ 

parameters  annotated  on  critical  technical 
characteristics  matrix? 

(2)  Reports  attesting  to  this  identified  in  _ 

Appendix  A  -  Bibliography? 

(3)  Are  critical  software  technical  _ 

parameters  addressed  by  developmental  test  and 
evaluation  annotated  on  the  Critical  Technical 
Parameters  Matrix  in  Part  I? 

c.  Future  Developmental  Test  and  Evaluation. 

(1)  Are  developmental  tests  designated  _ 

which  will  demonstrate  test  item  safety; 
supportability  (i.e.,  verify  and  validate  technical 
manuals  and  support  packages)  and  that 
specifications  are  met? 

(2)  Are  survivability/lethality  testing  as  _ 

well  as  those  tests  addressing:  E^  (Electromagnetic 
Environment  Effects)  conventional  weapon  effects, 

ECM,  ECCM,  initial  nuclear  weapon  effects,  advanced 
technology  survivability,  and  NBC  contamination 
identified? 

(3)  Are  test  plans  and  strategies  to  _ 

validate  the  manufacturing  process  identified? 

(4)  Are  the  following  areas  addressed  _ 
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throughout  the  DT&E ; 

RAM  _ 

Survivability  _ 

Electromagnetic  Capability  _ 

Human  Factors  _ 

System  Safety  _ 

Health  Hazards  _ 

Environment  _ 

Integrated  Logistical  Support  _ 

(5)  Is  each  test  presented  in  the  following  _ 

format:  Configuration  Description;  DT&E 
Objectives;  DT&E  Events,  Scope,  Basic  Scenario, 

and  Limitations? 

(6)  Are  the  differences  between  the  system  _ 

to  be  tested  and  objective  system  stated  for  each 
test? 


(7)  Are  the  resources  required  for  each  _ 

test  identified  in  PART  V? 

(8)  Are  test  and  evaluation  related  exit  _ 

criteria  identified  in  the  Acquisition  Decision 
Memorandum  (ADM) ,  addressed? 

(9)  Are  test  limitations  that  significantly  _ 

affect  the  developmental  evaluation  discussed  to 
include  software  developmental  testing  or  those 
developmental  tests  which  will  incorporate  the 
systems  embedded  software? 

(a)  Configuration  Management: 

Have  the  differences  between  software  _ 

functional  capabilities  of  the  system  to  be 
tested  and  those  of  the  objective  system  been 
identified? 

(b)  DT&E  Objectives: 

Have  software  test  objectives  _ 

for  this  phase  of  testing  been  stated? 

Has  the  method  for  software  _ 

evaluation  been  discussed? 

(c)  DT&E  Events,  Scope  of  Testing  and 
Basic  Scenarios: 

Have  the  key  planned  software  development  _ 

events  been  identified? 
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Is  there  a  discussion  of  the  analysis,  _ 

simulations,  subsystem  tests,  and  testbeds  which 
are  to  be  used  in  determining  if  software  DT&E 
objectives  are  met? 

Is  there  a  discussion  on  software  test  _ 

limitations  that  may  significantly  affect  the 
evaluator's  ability  to  draw  conclusions  and  make 
recommendations  concerning  software  technical 
parameters? 

d.  Live  Fire  Test  and  Evaluation. 

(1)  Overall  LFT&E  strategy  reflected?  _ 

(2)  LFT&E  issues  identified?  _ 

(3)  Required  levels  of  system  vulnerability  _ 

/lethality  reflected? 

(4)  Management  of  LFT&E  program  identified?  _ 

(5)  LFT&E  schedule  reflected?  _ 

(6)  Funding  identified?  _ 

(7)  Test  plans  identified?  _ 

(8)  Requirements  reflected?  _ 

(9)  Related  prior  and  future  LFT&E  efforts  _ 

identified? 

(10)  Evaluation  plan  identified?  _ 

(11)  Shot  selection  process  reflected?  _ 

(12)  Major  test  limitations  identified?  _ 


Part  IV.  Operational  Test  and  Evaluation  (OT&E)  Outline 
a.  Operational  Test  and  Evaluation  Overview. 

(1)  Relationship  between  program  schedule,  _  _ 

etc.,  and  system  requirements,  operational  issues, 

etc.,  reflected? 

(2)  OT  evaluation  identified?  _  _ 

(3)  DT  to  be  used  as  part  of  operational  _  _ 

evaluation  identified? 

(4)  Simulations /models  that  will  be  used  to  _  _ 
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augment  OT&E  reflected? 

b.  Critical  Operational  Issues. 

Approved  critical  operational  issues  listed?  _ 

Reference  made  to  approved  COICs  in  _ 

Appendix  A? 

c.  Operational  Test  and  Evaluation  to  Date. 

(1)  Each  phase  of  completed  OT&E  reflected?  _ 

(2)  System  tested  identified?  _ 

(3)  Summary  of  testing  that  occurred  _ 

reflected? 

(4)  Is  a  summary  of  what  has  been  learned  _ 

as  a  result  of  OT&E  about  the  hardware/software 
maturity  been  discussed? 

(5)  Are  the  differences  between  the  tested  _ 

hardware/ software,  hardware /software  planned  for 

the  current  phase,  and  hardware /software  to  be 
deployed  and  the  importance  of  these  differences 
been  discussed? 

d.  Future  Operational  Test  and  Evaluation. 

Evaluations/assessments  listed  as  well  as 
tests?  _ 

(1)  Configuration  Description 

Are  differences  described  between  tested  _ 

system  and  the  system  to  be  fielded? 

Is  the  extent  of  integration  with  other  _ 

systems  reflected? 

Is  the  system  characterized?  _ 

Has  the  software  and  hardware  configure-  _ 

tion  for  each  test  been  identified? 

Has  the  degree  to  which  test  results  _ 

from  this  configuration  represent  performance 
of  the  deployed  system  been  identified? 

(2)  OT&E  Objectives 

Are  test  objectives  including  the  critical  _ 

operational  issues  to  be  addressed  ty  each  phase 
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of  OT&E  and  the  decision  milestone (s)  stated? 

If  a  beyond  LRIP  decision  is  being  _ 

supported  are  test  objectives  that  examine  all 
areas  of  operational  effectiveness  and  suitability 
reflected? 

Has  the  relationship  between  OT&E  objectives  _ 

and  software  characteristics  which  affect 
critical  operational  issues  been  addressed? 

(3)  OT&E  Events,  Scope  of  Testing,  and  Scenarios 

Scenarios  summarized?  _ 

Events  to  be  conducted  identified?  _ 

Type  of  resources  to  be  used  reflected?  _ 

Simulation (s) /models  to  be  employed  _ 

identified? 

Type  of  representative  personnel  who  _ 

will  operate  and  maintain  the  system  reflected? 

Status  of  the  logistic  support  reflected?  _ 

Operational  and  maintenance  documentation  _ 

that  will  be  used  identified? 

Environment  under  which  the  system  is  _ 

to  be  employed  and  supported  during  testing 
reflected? 

Planned  sources  of  information  _ 

reflected? 

Has  the  relationship  between  software  _ 

functions  being  tested  and  test  scenarios  been 
discussed? 

Have  load  levels  to  be  used  and  their  _ 

relationship  to  the  required  operational  environ¬ 
ment  been  identified? 

(4)  Limitations 

Are  test  limitations  discussed  that  may  _ 

impact  the  resolution  of  affected  critical 
operational  issues? 

Are  critical  operational  issues  affected  _ 

indicated  in  parenthesis  after  each  limitation? 
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Have  the  identification  of  any  factors 
which  may  inhibit  realistic  OT  of  the  hardware/ 
software  been  identified? 

Have  constraints  been  identified  along 
with  their  impact  on  critical  operational  issues 
which  impose  on  software  maturity  or  availability 
of  resources  and  simulators? 


Part  V.  Test  and  Evaluation  Resource  Summary 

Is  a  summary  of  all  key  T&E  resources 
(government  and  contractor)  provided? 

Are  Major  Range  and  Test  Facility  Base 
resources  identified? 

a.  Test  Articles. 

1)  Are  actual  number  and  timing 
requirements  listed? 

(2)  Are  key  subsystems  to  be  tested 
separately  and  their  quantities  identified? 

(3)  Are  prototype,  development  pre- 
production  or  production  model  use  identified? 

b.  Test  Site  and  Instrumentation. 

(1)  Are  specific  test  range/facility  needs 
identified? 

(2)  Are  planned  test  range/ facility  needs 
identified  as  compared  with  existing  and 
programmed  capabilities? 

(3)  Are  new  instrumentation  acquisitions 
specified? 

c.  Test  Support  Equipment. 

(1)  Is  specifically  acquired  equipment 
identified? 

(2)  Are  unique/special  calibration 
requirements  indicated? 

d.  Threat  Systems /Simulators. 

(1)  Type/nximber/availability  identified? 

(2)  Are  requirements  identified  as  compared 
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with  available  and  projected  assets  and  their 
capabilities? 

(3)  Major  shortfalls  identified? 

(4)  Use  Accredited? 

e.  Test  Targets  and  Expendables. 

(1)  Type/number/availability  identified 
for  each  phase  of  testing? 

(2)  Major  shortfalls  identified?  _ 

(3)  Threat  targets  for  LFTE  identified?  _ 

(4)  Threat  munitions/systems  for  LFT  _ 

identified? 

f.  Operational  Force  Test  Support, 

Type  and  timing  of  aircraft  flight  hours,  _ 
etc.,  identified  for  each  phase? 

g.  Simulations,  Models  and  Testbeds. 

(1)  System  simulations  required  identified  _ 

for  each  phase? 

(2)  Rationale  for  usage/application  _ 

explained? 

(3)  Accreditation  Plan  prepared?  _ 

h.  Special  Requirements. 

Significant  non- instrument at ion  capabilities  _ 
and  resources  discussed? 

i.  Test  and  Evaluation  (T&E)  Funding  Requirements. 

(1)  FY  and  appropriation  line  number  _ 

reflected? 

(2)  Funding  required  to  pay  direct  costs  _ 

identified? 

(3)  Funding  currently  appearing  in  those  _ 

lines  indicated? 

(4)  Major  shortfalls  identified?  _ 

j .  Manpower/ Personnel  Training 
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Limitations  that  affect  test  execution 
identified? 

Appendix  A  -  Bibliography 

Reports  documenting  developmental  and 
operational  T&E  reflected? 

Appendix  B  -  Acronyms 

Appendix  C  -  Points  of  Contact 
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Chapter  1 
Introduction 

Section  I 
General 

1-1.  COIC  Objectives 

Critical  Operational  Issues  and  Criteria  (COIC)  provide 
operational  focus  for  materiel  and  automated  information  system 
development/acquisition  milestone  decisions,  and  serve  to 
prioritize  the  undergirding  operational  evaluation  effort.  COIC 
development  and  processing  objectives  are  to  deliver  an  approved 
set  of  COIC  which: 

a.  Provide  operationally  meaningful  and  relevant  concerns  and 
standards  key  to  the  Milestone  III  (full  production)  decision  to 
focus  the  supporting  independent  operational  evaluation  and 
foster  coordination  among  PM/MATDEV,  CBTDEV,  and  operational 
evaluator. 

b.  Are  available  in  a  timely  manner  to  support  the  Test  and 
Evaluation  Master  Plan  (TEMP)  process. 

c.  Conform  to  the  needs  of  the  materiel  acquisition  decision 
review  members  and  authorities,  as  well  as,  the  expectations  of 
COIC  and  TEMP  approval  authorities. 

1-2.  Background 

a.  Prior  to  1985,  COIC,  even  though  developed  by  the  combat 
developer  (CBTDEV),  were  a  part  of  the  operational  evaluation 
planning  process,  wherein  the  operational  issues  addressed  each 
component  of  operational  effectiveness  (e.g.,  equipment, 
software,  and/or  system  mission  capability,  survivability, 
interoperability,  etc.)  and  operational  suitability  (e.g., 
reliability,  availability,  and  maintainability  (RAM),  logistics 
supportability,  training,  manpower  and  personnel  integration 
(MANPRINT),  safety,  etc.)  as  applicable.  The  CBTDEV  designated 
with  an  asterisk  (*),  from  among  the  total  body  of  operational 
issues,  those  considered  to  be  critical.  While  some 
consideration  was  obviously  given  to  critical  issue  selection 
(regulations  mandated  many  areas  as  critical),  no  specific 
consideration  was  given  to  the  designation  of  critical  criteria 
and  all  were  therefore  considered  critical.  This  produced  COIC 
sets  with  numerous  issues  and  criteria. 

b.  In  September  1985,  after  several  "lessons  learned" 
experiences  with  systems  undergoing  acquisition,  HQDA  changed  its 
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COIC  philosophy.  COIC  were  to  be  few  in  number,  encompass  the 
total  system,  focus  on  the  system  mission,  be  operationally 
relevant,  and  be  realistic  (to  system  maturity)  for  the  supported 
decision.  They  were  not  to  deal  with  the  myriad 
elements /components  of  operational  effectiveness  and  suitability. 
It  was  thus  hoped  that  a  system  capable  of  satisfying  its 
essential  operational  needs /expectations  would  have  a  fair  chance 
of  moving  forward  in  the  acquisition  process.  The  1985  COIC 
philosophy  continues  today,  with  further  lessons  learned  applied 
to  mature  the  process. 

1-3.  Philosophy  and  content 

a.  Philosophy.  Critical  Operational  Issue  and  Criteria 
(COIC)  are  those  decision  maker  key  operational  concerns,  with 
bottom  line  standards  of  performance,  which  if  satisfied,  signify 
the  system  is  operationally  ready  to  proceed  into  full  production 
during  the  acquisition  decision  (normally  Milestone  III  but  may 
be  Engineering  Change  Proposal  [ECP]  or  Modification  Work  Order 
[MWO]  authorization  decision  for  modifications).  COIC  are  not 
pass/fail  absolutes  but  are  show  stoppers  such  that  a  system 
falling  short  of  the  criteria  will  not  proceed  into  full 
production  unless  convincing  evidence  of  its  operational 
effectiveness  and  suitability  is  provided  the  decision 
makers/authorities . 

b.  Focus  and  timing.  COIC  are  initially  prepare  and  approved 
for  inclusion  in  the  Test  and  Evaluation  Master  Plan  (TEMP) 
approved  prior  to  Milestone  I.  These  initial  COIC  are  based  on 
the  Mission  Needs  Statement  (MNS),  initial  Operational 
Requirement  (ORD)  (Functional  Description  (FD)  for  Automated 
Information  Systems  (AIS),  and  initial  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  with  other  documentation  when 
needed.  The  COIC  are  updated  and  approved  based  on  the  updated 
ORD  and  COEA  for  inclusion  in  the  TEMP  approved  for  the  Milestone 
II.  COIC  continually  focus  on  the  full  production  decision; 
therefore  revision  subsequent  to  Milestone  II  is  only  necessary 
for  significant  program  redirection,  preplanned  block  upgrades, 
and  other  modification  responding  to  revised  operational 
requirement.  The  issues  will  be  based  on  the  MNS  and  should 
remain  stable  during  the  acquisition  process.  The  criteria 
reflect  the  maturity  of  the  operational  requirement  (ORD,  FD,  and 
COEA);  therefore,  they  may  be  soft  Initially  (Milestone  I  TEMP) 
but  will  be  firm  standards  of  performance  for  the  Milestone  II 
TEMP.  Performance  exit  criteria  with  appropriate  operational 
considerations  may  be  used  to  guide  the  intermediate  milestone 
decisions  (e.g..  Milestone  II  and  Low  Rate  Initial  Production 
(LRIP)).  Such  exit  criteria  will  be  documented  in  the  TEMP  but 
not  as  part  or  the  COIC. 
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c.  Structure.  COIC  are  prepared  in  sets  which  include  the 
issues;  for  each  issue,  a  scope,  appropriate  criteria,  and 
rationale;  and  applicable  notes. 

(1)  Critical  Operational  Issue  (COI).  Those  key 
operational  concerns,  expressed  as  questions  which,  when  answered 
completely  and  affirmatively,  signify  that  a  system  or  materiel 
change  is  operationally  ready  to  transition  to  full  production. 

(2)  Scope  for  the  issue.  A  statement  of  the  operational 
capabilities,  definitions,  and  conditions  which  focus  the  issue 
and  guide  its  evaluation. 

(3)  Criteria  for  the  issue.  Those  measures  of  performance 
which,  when  achieved,  signify  that  the  issue  has  been  satisfied 
for  the  milestone  supported. 

(4)  Rationale  for  the  criteria.  Justification  for  COIC 
criteria  and  an  audit  trail  of  their  link  to  the  operational 
requirement  (MNS/ORD/Required  Operational  Capability  [ROC]/FD  and 
the  COEA) . 

(5)  Notes  for  the  COIC.  Both  mandatory  and  system  peculiar 
notes  apply.  The  mandatory  notes  are  modified  to  be  appropriate 
for  the  system. 

Section  II 

Organizational  Functions 
1-4 .  General 

The  functions  of  key  HQDA  Staff  and  Army  components  relative  to 
COIC  are  outlined  below.  NOTE:  The  parenthetical  ()  expression 
following  a  listed  function  identifies  the  staff  element, 
agency/activity,  or  subordinate  element  which  performs  the 
function. 

1-5.  Headquarters,  Department  of  the  Army  (HQDA) 

a.  Establishes  and  administers  Army  T&E  policy,  including 
COIC  policy.  (DUSA(OR)) 

b.  Develops  and  coordinates  Army  COIC  policy  and  procedures 
for  materiel  systems  and  Theater/Tactical  (T/T)  Information 
Mission  Area  (IMA)  systems.  Approves  and  distributes  COIC 
procedures  for  these  systems.  (DCSOPS,  ADCSOPS-FD/DAMO-FDR) 

c.  Develops  and  coordinates  Army  COIC  policy  and  procedures 
for  S/SB  IMA  systems.  Approves  and  distributes  COIC  procedures 
for  these  systems.  (DISC4,  VDISC4/SAIS-AE) 
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d.  Incorporates  appropriate  COIC  policy  and  procedures  in 
Army  regulations  and  pamphlets  governing  T&E,  materiel 
acquisition,  and  operational  need/requirements  definition  and 
documentation.  (TEMA,  SARDA,  and  DCSOPS) 

e.  Provides  the  interface  between  DA  and  OSD  on  T&E  policy 
matters,  including  COIC  policy  and  system  specific  T&E 
documentation  (TEMP,  Operational  T&E  [OT&E]  Plan  [TEP],  OT&E 
reports,  and  DOT&E  reports  to  Congress).  (DUSA(OR) /TEMA) 

f.  Maintains  and  distributes  a  schedule  reflecting  due  dates 
for  TEMP  delivery  to  HQDA  and  OSD  for  approval  (included  are  last 
approval  date(s)  for  the  TEMP  and  the  following  documents  which 
support  TEMP  submission:  STAR,  MNS,  ORD/ROC,  and  COIC).  (TEMA) 

g.  Distributes  DA  and  OSD  responses  (approval,  disapproval, 
or  comments)  on  Army  TEMPs  -  provides  copies  of  those  dealing 
with  COIC  to  DCSOPS,  DISC4,  and  TRADOC.  (TEMA) 

h.  Approves  COIC  for  inclusion  in  materiel  and  T/T  IMA 
systems  Milestone  II  TEMPs  which  require  DA  or  OSD  approval. 
Included  are  TEMPs  for  ACAT  I  and  II  materiel  systems,  T/T  MAISRC 
IMA  systems,  ACAT  III  and  IV  materiel  systems  and  T/T  IMA  systems 
on  the  OSD  T&E  oversight  list,  other  systems  chosen  by  DA,  and 
developmental  modifications  to  these  systems.  (ADCSOPS-FD) 

i.  Approves  COIC  for  all  Category  2  through  5  S/SB  IMA 
systems.  (VDISC4) 

j.  Coordinates  and  staffs  COIC  within  HQDA,  schedules  and 
coordinates  COIC  approval  briefings  within  DCSOPS  and  DISC4,  and 
distributes  approved  COIC.  (DAMO-FDR/SAIS-AE) 

k.  Reviews  TEMPs  submitted  to  DA  for  approval  for  inclusion 
of  DA  approved  COIC.  (DAMO-FDR/SAIS-AE) 

1-6.  Combat  Developer  (CBTDEV) 

a.  Develops  COIC  for  assigned  materiel  systems  (ACAT  I 
through  IV)  and  T/T  IMA  systems  in  coordination  with  the 
operational  evaluator,  appropriate  COEA  analyst,  Program/Project/ 
Product  Managers  (PM),  and  materiel  and  software  developers. 
(TRADOC  School/Centei/  Subordinate  Command  CBTDEV  or  TRADOC 
System  Manager  (TSM) ) . 

b.  Coordinates  COIC  for  assigned  materiel  and  T/T  IMA  systems 
with  Operational  Test  and  Evaluation  Command  (OPTEC),  the 
PM/MATDEV,  software  developer  (SFTDEV),  DAMO-FDR,  and  others  as 
required  to  establish  an  approval  recommendation.  (HQ  TRADOC 
System  Staff  Officer  (TRASSO)  for  those  systems  requiring  DA/OSD 
TEMP  approval;  School/  Center  CBTDEV  for  others) 
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c.  Approves  and  distributes  COIC  for  assigned  ACAT  III  and  IV 
materiel  and  Category  5  T/T  IMA  systems  not  on  the  OSD  T&E 
oversight  list  (includes  developmental  modifications  to  these 
systems).  (TRADOC  Center/School  Commander,  Commandant,  or 
Assistant  Commandant) 

d.  Approves  and  distributes  COIC  for  inclusion  in  the 
Milestone  I  TEMP  (includes  program  redirection  between  Milestones 
I  and  II)  for  those  assigned  materiel  and  T/T  IMA  systems 
requiring  DA/OSD  TEMP  approval  (Includes  developmental 
modifications  to  these  systems).  (HQ  TRADOC  DCSCD) 

e.  Approves  and  submits  COIC  to  HQDA  (DAMO-FDR) ,  through 
Commander  OPTEC  (CSTE-ZA),  for  approval  by  ADCSOPS-FD  for 
Inclusion  in  the  Milestone  II  TEMP  for  those  materiel  and  T/T  IMA 
systems  requiring  DA/OSD  TEMP  approval. 

f.  Prepares  and  briefs  the  ORD-COIC  Crosswalk  Horse  Blanket 
to  obtain  ADCSOPS-FD  approval  of  COIC  for  which  he  is  the 
approval  authority.  Appendix  E  discusses  preparation  of  the 
Horse  Blanket.  (TRADOC  Center/School/Subordinate  Command  CBTDEV 
or  TSM) 


g.  Attends  the  DA  ADCSOPS  COIC  approval  briefings.  (HQ 
TRADOC  TRASSO,  TSM,  and  TRADOC  Center /School  Subordinate  Command 
CBTDEV) 


h.  Assists  the  PM/MATDEV/SFTDEV  during  development  of  per- 
performance  exit  criteria  by  providing  applicable  operational 
considerations.  (TRADOC  Center/School  CBTDEV  or  TSM) 

1-7.  Functional  Proponent  for  Strategic  and  Sustaining  Base 
(S/SB)  Information  Management  Area  (IMA)  Systems 

a.  Develops  COIC  for  assigned  Category  2  through  5  S/SB  IMA 
systems  in  coordination  with  the  operational  evaluator  and  the 
PM/SFTDEV. 

b.  Submits  approved  COIC  to  HQDA  (SAIS-AE)  for  VDISC4 
approval . 

c.  Prepares  COIC-MNS  relationship  briefing  and  briefs  to 
VDISC4  to  obtain  approval. 

1-8.  Operational  Test  and  Evaluation  Command  (OPTEC) /Operational 
Evaluation  Command  (OEC) 

a.  Advises  CBTDEV  proponent,  IMA  Functional  Proponent  (FP), 
and  HQDA  whether  COIC  are  testable,  measurable,  or  otherwise 
evaluatable  and  provides  appropriate  structure  for  the 
operational  evaluation.  (OPTEC/OEC) 
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b.  Reviews  proposed  COIC  during  CBTDEV  proponent/FP 
development  and  approval  processing.  (OPTEC/OEC) 

c.  Endorses  COIC  from  CBTDEV  to  HQDA.  (CG  OPTEC) 

d.  Participates  during  DA  (ADCSOPS-FD  and  DISC4)  approval 
briefings.  (OPTEC/OEC) 

e.  Includes  TRADOC  approved  COIC  in  Test  and  Evaluation  Plan 
(TEP)  and  Part  IV  of  the  TEMP.  (OEC) 

f.  Assists  the  MATDEV  and  CBTDEV  in  formulation  of  Milestone 
(MS)  II  and  Low  Rate  Initial  Production  (LRIP)  performance  exit 
criteria.  (OPTEC/OEC) 

1-9.  Materiel  Developer  (MATDEV) 

a.  Advises  CBTDEV/FP  whether  COIC  criteria  are  consistent 
with  contract  specifications  and  reflect  appropriate  materiel  and 
software  maturity  expectations  at  the  full  production  decision 
point. 

b.  Reviews  proposed  COIC  during  CBTDEV/FP  development  and 
approval  process  and  participates  during  DA  (ADCS0PS-FD/DISC4 ) 
approval  briefings. 

c.  Incorporates  complete  set  of  approved  COIC  in  TEMP  Part 
IV,  Paragraph  4B. 

d.  Formulates,  in  conjunction  with  the  lOE  and  CBTDEV, 
performance  oriented  exit  criteria  for  MS  II  and  LRIP  decision 
reviews  and  document  as  appropriate  in  the  TEMP. 

1-10.  Software  Developer  (SFTDEV) 

a.  Advises  CBTDEV/FP  whether  COIC  criteria  are  consistent 
with  contract  specifications  and  reflect  appropriate  materiel  and 
software  maturity  expectations  at  the  full  production  decision 
point . 

b.  Reviews  proposed  COIC  during  CBTDEV/FP  development  and 
approval  process  and  participates  during  DA  (ADCSOPS-FD/DISC4 ) 
approval  briefings. 

c.  Incorporates  complete  set  of  approved  COIC  in  TEMP  Part 
IV,  Paragraph  4B. 
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Chapter  2 

COIC  Development  and  Approval  Processes 


Section  I 
General 


2-1.  COIC  applicability 

a.  Materiel  acquisition  programs.  COIC  are  required  for  all 
materiel  system  acquisition  programs,  developmental  and 
nondevelopmental .  COIC  are  also  required  for  system  preplanned 
block  improvements  and  materiel  changes  engendered  by  changes  to 
the  requirements  document  (ROC/ORD). 

b.  Information  Mission  Area  (IMA)  systems  acquisitions.  COIC 
are  required  for  all  Category  2  through  5  automated  information 
system  (AIS)  acquisitions  in  Information  Mission  Area  (IMA). 

COIC  are  also  required  for  preplanned  block  improvements  for 
these  systems  and  for  software  changes  in  response  to  changes  to 
the  requirements  document  (MNS).  NOTE;  Category  1  IMA  systems 
are  handled  as  major  materiel  system  acquisitions;  Category  6  IMA 
systems  are  exempt  from  TEMP  development,  and  thus  COIC, 
requirements . 

c.  Non-acquisition  decision  test  and  evaluation.  COIC  are 
not  applicable  to  other  than  operational  test  and  evaluation 
supporting  materiel  and  IMA  systems  acquisition  decision  .  Thus, 
TRADOC  Force  Development  Evaluation  (FDEV)  including  Force 
Development  Test  and  Experimentation  (FDTE)  and  the  Concept 
Evaluation  Program  (CEP)  do  not  require  COIC.  OPTEC  may  use  data 
from  such  tests  and  evaluations  as  another  data  source  for 
evaluation  satisfaction  of  the  COIC.  TRADOC  and  other  CBTDEV  and 
training  developers  use  a  set  of  Operational  Issues  and  Criteria 
(OIC),  similar  to  Additional  Operational  Issues  and  Criteria 
(AOIC)  described  in  paragraph  3-4  of  this  Part,  for  these 
evaluations. 

2-2.  COIC  and  the  materiel  acquisition  process 
The  relationship  of  COIC  to  the  materiel  acquisition  process  is 
graphically  depicted  in  Figure  2-1.  Similar  relationships  apply 
for  the  AIS  acquisition  process. 

*******insERT  FIGURE  2-1******** 

a.  COIC  development  and  purpose.  The  CBTDEV  proponent/FP 
produces  COIC  as  a  basis  for  subsequent  determination  by 
appropriate  DoD/Army  decision  authority  that  a  system  is 
operationally  ready  for  full  production.  Critical  Operational 
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Issues  (COI),  based  on  the  Mission  Need  Statement  (MNS),  are 
developed  for  inclusion  in  the  MS  I  TEMP.  In  that  they  are  based 
on  the  MNS,  they  seldom  change.  COI  are  qualified  by  criteria 
focused  on  the  full  production  (MS  III)  decision.  These  are 
based  on  the  Operational  Requirements  Document  (ORD) ,  Cost  and 
Operational  Effectiveness  Analysis  (COEA) ,  and  other  studies  that 
occur  throughout  the  process.  They  are  thus  subject  to  change 
and/or  progressively  Increasing  stringency  as  the  system  matures. 
COIC  are  included  in  the  system  TEMP,  used  to  guide  the 
MATDEV/SFTDEV  formulation  of  performance  oriented  milestone 
decision  review  exit  criteria,  and  form  the  basic  structure  of 
the  evaluation  effort. 

b.  COIC  coordination.  The  CBTDEV  Proponent/TRASSO/FP  staff 
and  coordinate  COIC  at  applicable  command  levels  (e.g.,  TRADOC 
and  DA)  with  the  MATDEV  and  with  the  operational  evaluator. 

c.  COIC  approval  authorities. 

(1)  The  ADCSOPS-FD  approves  ACAT  I,  II,  and  OSD  T&E  Oversight 
system  COIC  for  inclusion  in  the  MS  II  TEMP  and  all  COIC 
revisions  thereafter  to  focus  the  operational  evaluation 
supporting  the  full  production  (Milestone  III)  decision  for 
materiel  systems  and  theater/tactical  Major  Automated  Information 
Systems  (MAIS). 

(2)  The  VDISC4  approves  COIC  at  the  DA  level  for  S/SB  IMA 
systems . 

(3)  CBTDEV  Command  (HQ  TRADOC)  approves  ACAT  I,  II,  and  OSD 
T&E  Oversight  system  COIC  for  Inclusion  in  the  MS  I  TEMP,  changes 
to  these  COIC  resulting  between  MS  I  and  MS  II  not  intended  for 
the  MS  II  TEMP  (e.g.,  before  final  ORD  approval),  and  COIC  for 
all  ACAT  III  and  IV  materiel  and  Category  5  Theater/Tactical  AIS 
not  on  the  OSD  T&E  Oversight  List. 

2-3.  Synchronized  ORD,  COIC,  and  TEMP  schedules 
Tables  2-1  through  2-3  provide  COIC  process  synchronization 
schedules  for  materiel  acquisition  and  IMA  systems.  They  should 
be  used  by  PM/MATDEV/SFTDEV,  operational  evaluator,  and  CBTDEV/FP 
for  planning  and  management  of  these  events  during  acquisition 
programs  to  minimize  the  chance  of  milestone  decision  review 
(MDR)  delay. 

a.  Materiel  acquisition  and  T/T  IMA  systems  MS  I  and  II. 

Table  2-1  provides  COIC  processing  deadlines  synchronized  with 
those  of  the  ORD  and  TEMP  events.  For  ACAT  ID  programs,  add  15 
days  to  the  figures  indicated  for  ACAT  IC. 

*********insERT  TABLE  2-1*************** 
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b.  Materiel  acquisition  and  T/T  IMA  system  post  MS  II  COIC 
changes.  Table  2-2  provides  a  similar  schedule  with  adjustments 
which  apply  in  the  event  COIC  require  update  after  MS  II.  NOTE 
that  timing  is  keyed  to  DA  approval  of  the  TEMP  and  not  a 
specific  MDR. 

*********INSERT  TABLE  2-2**************** 

c.  Strategic/sustaining  base  (S/SB)  IMA  systems.  Table  2-3 
provides  COIC/TEMP  deadlines  for  S/SB  IMA  systems  keyed  to  MDR  I. 
Subsequent  updates  would  follow  similar  schedules  synchronized 
with  TEMP  delivery  suspense  or  decision  milestone  schedule  as 
appropriate. 

*********INSERT  TABLE  2-3**************** 

Section  II 

COIC  Process  for  ACAT  I/II  Materiel,  Theater/Tactical  (T/T) 

MAISRC  AIS,  OSD  T&E  Oversight,  and  Other  Systems  Selected  for 
HQDA  ADCSOPS-FD  Approval  (Figures  2-2  through  2-7) 

2-4.  Forward 

Detailed  procedures  for  development,  review,  and  approval  of  COIC 
for  ACAT  I/II  materiel,  T/T  MAISRC  AIS,  OSD  T&E  Oversight,  and 
other  systems  selected  for  approval  by  HQDA  (ADCSOPS-FD)  are 
described  on  the  following  pages  in  the  form  of  descriptive 
paragraphs  on  one  page  and  corresponding  flow  chart  elements  on 
the  facing  page.  Procedures  also  apply  to  COIC  changes 
necessitated  by  developmental  modifications  to  these  system. 
Reflected  Not  Later  Than  (NLT)  dates  are  minimum  deadlines  to 
allow  on  schedule  delivery  of  the  TEMP.  Managers  should 
therefore  establish  appropriate  internal  suspenses  and  effect 
early  submission  where  possible  to  ensure  compliance.  Dotted/ 
dashed  boxes  and  lines  in  the  figures  are  information 
events /blocks.  When  appropriate,  "NOTES"  are  added  to  highlight 
actions  called  for  in  the  paragraph  or  to  provide  some  other 
insight  into  the  action  required. 

2-5.  CBTDEV  proponent  drafts  COIC  for  MS  I  TEMP  (Figure  2-2) 

a.  Front  end  analysis.  The  process  begins  with  the  CBTDEV 
proponent  conducting  COIC  front-end  (comparative)  analysis.  This 
analysis  uses  as  its  base  the  MNS  approved  at  Milestone  0  and 
draws  from  the  subsequent  ORD  formulation  process.  Adu'tional 
considerations  Include,  but  are  not  necessarily  limited  to: 
baseline  intelligence  products  (or  the  System  Threat  Assessment 
Report  (STAR),  if  available);  system  critical  mission(s)  and 
function(s);  system  employment  and  sustainment  concepts;  similar 
system(s)  acquisition  experience;  studies/analyses  used  to 
justify  the  system  requirement;  the  (COEA),  when  available;  and 
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recent  COIC  approval  process  experiences  (approved  examples, 
questions,  rejections,  and  guidance). 

NOTE:  Object  of  front-end  analysis  is  to  "get  smart"  on  the 

system  so  that  COIC  can  be  properly  focused  and  accurately 
reflect  bottom  line  critical  mission  accomplishment  and 
sustainment  performance  standards. 

b.  Draft  COIC  for  MS  I  TEMP.  In  conjunction  with  submission 
of  the  draft  ORD  to  HQ  TRADOC  for  approval,  the  CBTDEV  proponent 
prepares  draft  COIC  focused  on  the  full  production  (MS  III) 
decision.  The  critical  issues,  being  based  on  the  MNS,  are 
unlikely  to  change  as  the  system  proceeds  through  the  acquisition 
phases.  Criteria,  on  the  other  hand,  may  initially  be  "soft" 
(i.e.,  lack  specificity) .  Detailed  preparation  considerations 
and  guidelines  are  found  in  Chapter  3  of  this  Part. 

********inseRT  figure  2-2*********** 

2-6.  COIC  coordination  and  approval  for  MS  I  TEMP  (Figure  2-3) 

a.  CBTDEV  proponent  coordination.  Following  receipt  of  the 
TRADOC  approved  ORD  (forwarded  to  HQDA  NLT  185  days  prior  to  MS 
I),  the  CBTDEV  proponent  makes  appropriate  adjustments  to  the 
draft  COIC  and  coordinates  them  with  the  MATDEV,  operational 
evaluator,  and  TRADOC  Analysis  Command  (TRAC).  Coordination  with 
the  MATDEV  will  ensure  synchronization  with  software  requirements 
definition  documentation,  RAM  (reliability,  availability,  and 
maintainability)  rationale  reporting,  and  system  specifications 
to  be  documented  in  a  request  for  proposal  (RFP) .  Coordination 
with  the  operational  evaluator  will  ensure  that  COIC  are 
testable,  measurable,  or  otherwise  evaluatable  and  provide  an 
appropriate  structure  for  the  operational  evaluation. 

Additionally  (Block  3A),  in  that  COIC  focus  on  the  full 
production  (MS  III)  decision,  the  MATDEV  with  CBTDEV  and  lOE 
assistance  system  formulates  performance  exit  criteria  for  the  MS 
II  (Development)  decision  review.  Coordination  with  TRAC  ensures 
appropriate  COEA  synchronization. 

NOTE:  A  discussion  of  the  relationship  of  performance  exit 

criteria  to  COIC  can  be  found  in  Section  IV,  this  chapter. 

b.  CBTDEV  proponent  submits  COIC  for  approval.  Not  later 
than  140  days  prior  to  MS  I,  the  CBTDEV  proponent  forwards  the 
coordinated,  final  draft  COIC  to  HQ  TRADOC  for  review  and 
approval.  Concurrently,  the  MATDEV  includes  the  draft  COIC  in 
the  draft  TEMP  to  maintain  currency  of  the  TEMP. 

c.  HQ  TRADOC  quality  check.  At  HQ  TRADOC,  the  TRASSO 
performs  a  final  quality  check  and,  if  necessary  (Block  5A), 
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revises  the  COIC  with  assistance  of  the  CBTDEV  proponent.  When 
required/  MATDEV  and/or  operational  evaluator  may  assist. 

d.  MS  I  TEMP  COIC  approval  staffing.  Following  DA  approval 
of  the  ORD  and  not  later  than  125  days  prior  to  MS  I,  with  a 
2-week  suspense/  the  TRASSO  formally  staffs  the  final  draft  COIC 
with  the  TRADOC  Materiel  Evaluation  Committee  (TMEC)/  HQ  OPTEC/ 
and  the  MATDEV.  Additionally/  the  TRASSO  effects  action  officer 
(A/0)  I'evel  coordination  with  DA  DCSOPS  (DAMO-FDR)  .  If  DA 
changed  ORD  during  approval/  TRASSO  takes  lead  to  effect  changes 
to  COIC/  as  necessary/  with  assistance  as  required  (Block  5A) . 

e.  COIC  approval  for  MS  I  TEMP.  The  TRADOC  DCSCD  approves 
COIC  NLT  MS  I  minus  95  days  and  provides  them  to  the  MATDEV  for 
inclusion  in  the  TEMP  as  paragraph  4B/  Part  IV.  If  he  directs 
changes/  the  process  loops  back  to  Block  5/  TRASSO  quality  check. 

NOTE:  The  obvious  pitfall  here  is  one  of  timing/  considering 

that  restaffing  could  take  as  much  as  two  additional  weeks  and 
thereby  potentially  negatively  impact  MS  I. 


***********INSERT  figure  2-3************* 

2-7.  MS  I  TEMP  approval  and  operational  requirement  maturation 
(Figure  2-4) 

a.  TEMP  approval  processing.  The  MATDEV  submits  ACAT  1/  II/ 
and  OSD  T&E  Oversight  system  TEMP  for  review  and  approval  to 
TEMA/DUSA(OR) /  65  days  in  advance  of  MS  1,  as  an  integral  part  of 
the  TEMP.  TEMA  forwards  the  approved  TEMP  to  OSD  for  review  and 
approval  45  days  in  advance  of  MS  I.  If  changes  are  directed  at 
either  DA  or  OSD  level  (Block  9A) /  an  ad  hoc  working  group 
(DCSOPS  (DAMO-FDR)  A/0/  TRASSO/  CBTDEV  proponent/  MATDEV/  and 
operational  evaluator)  resolves  the  changes  and  their  impacts 
(Block  9B).  The  TRADOC  DCSCD  reviews  and  approves  revised  COIC 
(Block  9C)  for  Incorporation  into  and  resubmission  of  the  revised 
TEMP  by  the  MATDEV  (Block  9D  to  Block  9).  An  approved  TEMP  is 
required  at  MS  I . 

b.  OPTEC  Early  Operational  Assessment  (EOA)  planning. 
Following  receipt  of  the  approved  TEMP/  OPTEC  incorporates  COIC 
into  the  TEP  for  Early  Operational  Assessment  (EOA)  and  uses 
them/  along  with  any  Decision  authority  approved  performance  exit 
criteria/  as  a  basis  for  finalization  of  Additional  Operational 
Issues  and  Criteria  (AOIC)  to  guide  the  test  and  evaluation 
effort.  Chapter  3/  this  Part  discusses  AOIC. 

c.  CBTDEV  identifies  required  COIC  revisions.  The 
operational  requirement  matures  as  a  system  progresses  through 
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its  acquisition  phases.  As  a  result,  revisions  to  COIC  may  be 
necessitated  to  ensure  that  they  continue  to  adequately  and 
accurately  represent  features/capabilities  critical  to  mission 
performance  and  provide  a  proper  focus  for  the  decision  process. 
Additionally,  the  CBTDEV  proponent  needs,  when  and  where 
possible,  to  restructure  criteria  to  reflect  a  greater  level  of 
specificity  (firmness)  than  that  found  in  the  "soft"  criteria  of 
the  initial  set.  The  CBTDEV  proponent  identifies  necessity  for 
change/update  by  actively  participating  in  and  maintaining  full 
cognizance  of  the  system's  developmental  progress.  First  among 
many  potential  sources  of  change  is  the  Acquisition  Decision 
Memorandum  (ADM),  which  documents  decisions  and  directives  of  the 
Milestone  Decision  Review  (MDR)  and  approves  the  system  concept 
baseline  (MS  I)  and  exit  criteria  for  the  next  MS.  Additional 
sources  include,  but  are  not  limited  to,  results  from  EUTE/EOA 
(Block  11),  emerging  results  from  the  COEA,  the  ORD 
refinement/revision  process  (to  culminate  in  HQ  TRADOC  approval 
230  days  in  advance  of  MS  II),  and  development  of  the  RAM 
Rationale  Report  and  materiel  specifications  for  the  RFP. 

***********inSERT  figure  2-4*********** 

2-8.  COIC  update  by  the  CBYDEV  for  MS  II  TEMP  (FIGURE  2-5) 

a .  CBTDEV  proponent  updates  COIC .  The  CBTDEV  proponent 
updates  COIC  as  necessary,  to  include  the  addition  of  "firm"  MS 
III  criteria  (i.e.  provides  a  specific,  usually  quantitative, 
performance  threshold).  Concurrently  (Block  13A),  he  works  with 
the  operational  evaluator  and  PM  in  the  formulation  of  LRIP 
and/or  MS  III  exit  criteria. 

NOTE;  Exit  criteria  for  MS  II  were  formulated  pre-MS  I,  approved 
during  MDR-I,  and  documented  in  the  ADM. 

b.  CBTDEV  proponent  COIC  coordination.  The  CBTDEV  proponent 
coordinates  draft  COIC  with  the  PM,  operational  evaluator,  and 
TRAC,  and,  not  later  than  185  days  in  advance  of  MS  II,  submits 
the  final  draft  COIC  to  HQ  TRADOC  for  review  and  approval. 
Concurrently,  the  MATDEV  includes  the  final  draft  COIC  in  the 
TEMP  for  TIWG  staffing. 

C.  HQ  TRADOC  quality  check  of  COIC.  At  HQ  TRADOC,  the  TRASSO 
performs  a  final  quality  check  and,  if  changes  are  necessary 
(Block  16A),  revises  the  COIC  with  the  assistance  of  the  CBTDEV 
proponent.  Additional  assistance,  when  required,  is  drawn  from 
the  PM  and/or  operational  evaluator. 

d.  CBTDEV  COIC  approval  staffing  for  MS  II  TEMP.  Not  later 
than  170  days  in  advance  of  MS  II,  HQ  TRADOC  (TRASSO)  formally 
staffs  the  final  draft  COIC  with  the  TMEC,  HQ  OPTEC,  and  the  PM. 
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Additionally,  the  TRASSO  effects  A/0  level  coordination  with  the 
DA  DCSOPS  (DAMO-FDR) .  DA  must  approve  the  revised  ORD  before 
this  coordination  may  proceed.  Since  submission  and  quality 
check  may  occur  before  ORD  approval,  changes  may  be  needed  to  be 
consistent  with  the  ORD.  If  this  is  the  case,  the  TRASSO  effects 
necessary  changes,  with  assistance  as  required  (Block  16A) . 

NOTE;  For  ACAT  ID  programs,  for  which  the  Under  Secretary  of 
Defense  (Acquisition)  is  the  milestone  decision  authority,  not 
later  than  dates  are  15  days  earlier  than  reflected  in  the  text 
or  on  flow  chart  elements  to  accommodate  Army  review  and  approval 
in  advance  of  the  Defense  Acquisition  Board  (DAB)  review. 

e.  CBTDEV  (HQ  TRADOC)  submits  COIC  for  MS  II  TEMP  to  HQDA. 

The  DCSCD,  HQ  TRADOC  approves  the  COIC  for  TRADOC,  not  later  than 
140  days  in  advance  of  MS  II,  and  submits  them  (Block  18A) 
through  the  Commanding  General  (CG)  OPTEC  for  endorsement  to  HQDA 
(DAMO-FDR)  for  ADCSOPS-FD  approval.  Advance  copies  are  provided 
to  HQDA  (DAMO-FDR),  the  PM,  Operational  Evaluation  Command  (OEC), 
appropriate  TRADOC  centers  and  schools,  and  others. 

************INSERT  figure  2-5*************** 

2-9.  COIC  approval  processing  for  MS  II  TEMP  (Figure  2-6) 

a.  CG  OPTEC  endorsement  of  MS  II  TEMP  COIC.  CG  OPTEC  concurs 
in  the  COIC  and  endorses  them  to  HQDA  (DAMO-FDR),  NLT  115  days  in 
advance  of  MS  II,  as  testable,  measurable,  or  otherwise 
evaluatable  and  capable  of  providing  an  appropriate  structure  for 
the  operational  evaluation  in  support  of  the  full  production  (MS 
III)  decision.  In  the  event  of  CG  OPTEC  nonconcurrence,  DCSCD 
TRADOC  (Block  19A)  directs  revision  (return  to  Block  16A)  or  opts 
to  allow  the  nonoccurrence  to  remain  an  issue  for  resolution  at 
the  DA  ADCSOPS-FD  decision  briefing. 

b.  MS  II  TEMP  COIC  approval  brief  preparations.  DAMO-FDR 
schedules  the  DA  ADCSOPS-FD  Director  pre-brief  and  the  ADCSOPS-FD 
decision  briefing.  The  CBTDEV  proponent  prepares  the  COIC-ORD 
cross-walk  "Horse  Blanket"  and  provides  it  to  HQ  TRADOC,  if 
required,  for  review  or  possible  pre-brief.  DAMO-FDR  also 
prepares  a  decision  package  for  the  ADCSOPS-FD,  taking  into 
account  COIC  experience,  CG  OPTEC  concurrence/nonconcurrence  in 
the  TRADOC  approved  COIC,  and  the  COIC-ORD  crosswalk  package. 

NOTE:  The  basic  content  of  the  "Horse  Blanket"  and  its  physical 
layout  are  depicted  within  a  dashed-line  box  on  the  flow  chart. 

A  detailed  description  of  the  "Horse  Blanket",  its  content,  and 
sample  questions  occurring  during  briefing  are  presented  in 
Section  V  of  this  chapter. 
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c.  COIC  approval  pre-brief.  A  COIC-ORD  Crosswalk  pre-brief 
(normally  1-2  days  before  the  formal  briefing  and  chaired  by  the 
ADCSOPS-FD  Integration  Director)  determines  readiness  of  the 
briefing  for  ADCSOPS-FD.  Changes  directed  at  this  stage  will 
normally  be  made  without  delaying  the  approval  briefing. 

d.  MS  II  TEMP  COIC  brief.  The  ADCSOPS-FD  reviews  and 
approves  COIC  via  the  COIC-ORD  Crosswalk  briefing  conducted  not 
later  than  100  days  in  advance  of  MS  II.  Coordinated  by 
DAMO-FDR/  briefing  participants  include  the  DA  DCSOPS  System 
Integrator  (SI)/  the  TRASSO/  and  representatives  of  the  CBTDEV 
proponent,  PM,  lOE,  TRADOC  System  Manager  (TSM),  DCSINT,  and 
others  as  needed.  The  TSM  or  Proponent  CBTDEV  /FP  briefs.  When 
required,  this  briefing  also  serves  as  a  joint  DA  ADCSOPS-FD,  CG 
OPTEC,  TRADOC  DCSCD  joint  meeting  to  address  unresolved 
conflicts,  e.g.,  CG  OPTEC  nonconcurrence  in  the  TRADOC  approved 
COIC.  If  minor  changes  to  the  COIC  are  directed,  they  are 
handled  by  the  DAMO-FDR  A/0  (Block  25A) .  If  the  changes  are 
significant,  however,  a  return  to  Block  16A  is  probable. 

e.  COIC  approval  for  MS  II  TEMP.  ADCSOPS-FD  approves  the 
COIC  and  forwards  them  to  the  TIWG  Chair  (MATDEV  or  PM)  for 
Inclusion  in  the  TEMP  not  later  than  95  days  in  advance  of  MS  II. 

**************inseRT  figure  2-6*************** 

2-10.  MS  II  TEMP  approval  and  COIC  updates  (Figure  2-7) 

a.  TEMP  approval  processing.  The  PM  submits  system  TEMP  for 
review  and  approval  to  TEMA/DUSA(OR) ,  not  later  than  65  days  in 
advance  of  MS  II.  TEMA  forwards  the  approved  TEMP  to  OSD  for 
review  and  approval  not  later  than  45  days  in  advance  of  MS  II. 

If  changes  are  directed  at  either  DA  or  OSD  level  (Block  27A) ,  DA 
DCSOPS  (DAMO-FDR)  A/0,  TRASSO,  CBTDEV  proponent,  MATDEV/ SFTDEV, 
and  operational  evaluator  resolve  the  changes  and  their  impacts 
(Block  27B) .  The  DCSCD  TRADOC  and  ADCSOPS-FD  HQDA  reviews  and 
approves  revised  COIC  (Block  27C)  for  MATDEV  inclusion  into  and 
resubmission  of  the  revised  TEMP  (Block  27D  to  Block  27). 

b.  OPTEC  lOT&E  preparations.  Following  receipt  of  the 
approved  TEMP,  OPTEC  incorporates  COIC  into  the  TEP  for  lOTE  and 
uses  them,  and  performance  exit  criteria,  as  a  basis  for 
development  of  Additional  Operational  Issues  and  Criteria  (AOIC) 
(see  chapter  3,  this  part)  necessary  to  the  evaluation  effort. 

c.  Post  MS  II  COIC  changes.  Post  MDR-II  (Development 
Approval),  the  CBTDEV  proponent  continues  to  actively  participate 
in  and  maintain  full  cognizance  of  the  system's  developmental 
progress.  In  that  firm  MS  III  COIC  were  approved  and  included  in 
the  TEMP  at  MS  II,  COIC  will  not  be  further  modified  unless 
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directly  affected  by  operational  requirements  changes  (e.g., 
necessitated  by  new/revised  threat  intelligence  information)  or 
program  restructure  (e.g./  performance  to  be  achieved  by  MS  III 
or  as  a  result  of  Block  Improvements).  If  revision  is 
necessitated,  the  CBTDEV  proponent  reenters  the  process  and 
completes  Blocks  13-28,  to  support  of  the  full  production 
decision  (MS  III).  Further,  in  the  event  an  ACAT  III  system 
becomes  designated  a  OSD  T&E  Oversight  system  after  MS  II,  the 
CBTDEV  proponent  reenters  the  process  and  completes  Blocks  15-28. 

d.  COIC  for  system  modifications.  Only  developmental 
modifications  require  COIC  (i.e.,  preplanned  upgrades  and  other 
modifications  responding  to  ORD  revisions).  For  such 
modifications,  a  TEMP  with  approved  COIC  is  a  key  element  of  the 
modification  proposal  approval  package.  These  elements  will 
guide  the  evaluation  and  decision  to  adopt  the  modification.  The 
CBTDEV  proponent  therefore  reenters  the  process  and  completes 
Blocks  13-27  in  support  of  the  modification  approval  (MS  IV) . 

DOD  5000.2  requires  MS  IV  to  approve  initiation  of  major 
modifications.  Paragraph  2-1,  this  chapter  provides  additional 
Information  on  modifications  requiring  COIC. 

NOTE:  When  changes  are  required  after  MS  II,  process  timing  is 

jteyed  to  days  in  advance  of  DA  approval  of  the  TEMP  as  per  Table 
2-3,  this  chapter. 

************INSERT  figure  2-7************ 

2-11.  Sample  final  COIC  coordination  and  approval  submission 
memorandums  for  those  requiring  HQDA,  ADCSOPS-FD  approval 

a.  Final  COIC  coordination  memorandum.  Figure  2-8  provides  a 
sample  final  coordination  memorandum  for  COIC  requiring  HQDA, 
ADCSOPS-FD_approval .  The  following  observations  are  made 
regarding  this  memorandum: 

(1)  This  coordination  is  the  basis  for  TRADOC,  DCSCD 
approval  and  submission  of  the  COIC  to  DA.  Only  CG  OPTEC  or 
ADCSOPS-FD  can  cause  TRADOC  to  revisit  the  COIC  once  approved  by 
the  TRADOC,  DCSCD.  The  TRADOC  Materiel  Evaluation  Committee 
(TMEC)  advises  whether  the  COIC  appropriately  reflect  the 
operational  requirements .  The  PM  advises  whether  the  COIC  are 
compatible  with  technology  and  contractual  documents.  OPTEC 
advises  whether  the  COIC  are  testable,  measurable  or  otherwise 
evaluatable.  DAMO-FDR  provides  DA  action  officer  coordination 
and  advise  confirming  latest  events  in  HQDA  relative  to  the 
system  and  COIC. 

(2)  This  is  the  last  coordination  with  the  PM  before  the 
COIC  approval  brief  to  ADCSOPS-FD.  CG  OPTEC  has  the  opportunity 
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to  confirm  concurrence  in  the  COIC  by  endorsing  the  TRADOC,  DCSCD 
approved  COIC  to  HQDA  for  approval.  The  DA  staff  have  the 
opportunity  to  concur  with  the  COIC  during  decision  package 
coordination. 

(3)  A  two  week  suspense  for  this  action  is  normal. 
Occasionally  shorter  or  longer  time  frames  may  be  available.  Two 
weeks  is  about  the  maximum  allowance  to  support  approval  within 
30  days  after  receipt  at  HQ  TRADOC  as  allowed  by  the  ORD#  COIC, 
and  TEMP  synchronization  schedule. 

(4)  Significant  changes  will  be  recoordinated  using 
expedited  procedures  (most  likely  electronic  malls  or  fax)  with 
short  turnaround  required. 

(5)  Any  significant  delay  during  the  coordination  will 
likely  jeapordize  the  TEMP  submission  schedule.  This  applies  to 
the  initial  as  well  as  follow-up  coordination. 

(6)  The  TRADOC  proponents  are  kept  Informed  regarding  the 
status  of  their  COIC  by  the  copy  furnished  distribution. 

****************INsERT  figure  2-8************ 

b.  COIC  submission  for  HQDA,  ADCSOPS-FD  approval  memorandum. 
Figure  2-9  provides  a  sample  COIC  approval  submisisslon 
memorandum.  The  following  observations  apply  to  this  memorandum: 

(1)  The  submission  is  through  CG  OPTEC  to  HQDA  (DAMO-FDR) 
in  keeping  with  the  guidance  that  only  CG  OPTEC  and  the  ADCSOPS- 
FD  can  cause  TRADOC  to  revisit  COIC  approved  by  DCSCD,  TRADOC. 

CG  OPTEC  endorsement  attests  to  the  COIC  being  testable, 
measurable,  or  otherwise  evaluatable  and  provide  structure  for 
operational  evaluation.  A  nonconcurrence  or  significant  comment 
should  be  exceptional  circumstance  since  OEC  has  been  evolved 
throughout  the  COIC  development  process  and  OPTEC  provided  its 
position  during  TRADOC  final  position  staffing. 

(2)  The  memorandum  recognizes  whether  TRADOC,  PM,  and  OPTEC 
agreement  was  reached  during  the  TRADOC  approval  process.  In 
those  cases  where  agreement  not  reached,  the  difference  of 
position  is  described. 

(3)  The  copy  furnished  distribution  keeps  key  players  in 
the  COIC  approval  briefing  informed  as  to  status  of  the  COIC.  In 
the  case  of  DAMO-FDR  and  the  TRADOC  proponent  this  distribution 
provides  an  advance  copy  of  the  COIC  to  support  preparation  for 
COIC  approval  brief. 

**********INSERT  FIGURE  2-9************ 
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Section  III 

Process  for  ACAT  III/IV  Materiel  and  Theater/Tactical  (T/T) 
Non-MAISRC  IMA  Systems  not  on  OSD  T&E  Oversight  List  (Figure  2-10 
through  2-14). 


2-12.  Forward 

Procedures  for  development/  review,  and  approval  of  COIC  for  ACAT 
III/IV  materiel  and  T/T  non-MAISRC  IMA  systems  not  on  the  OSD  T&E 
Oversight  List  are  similar  to  those  for  ACAT  I/II  materiel,  T/T 
MAISRC  IMA  systems,  and  OSD  T&E  Oversight  systems.  Primary 
differences  are  approval  authority  and  timing  (Figure  2-2,  this 
chapter) .  As  in  previous  section,  descriptive  paragraphs  appear 
on  one  page  and  corresponding  flow  chart  elements  on  facing  page. 
Process  also  apply  to  COIC  for  developmental  modifications  to 
these  systems.  Reflected  Not  Later  Than  (NLT)  dates  are  minimum 

deadlines  for  on  schedule  delivery  of  the  TEMP. _ Managers  should 

therefore  establish  appropriate  internal  suspenses  to  effect 
early  COIC  approval  and  TEMP  submission.  Dotted/dashed  boxes  and 
lines  are  information  events.  A  checklist  to  assist  in  the 
process  is  at  Section  V,  this  chapter.  When  appropriate,  "NOTES" 
are  added  to  highlight  actions  called  for  in  the  paragraph  or  to 
provide  some  other  insight  into  the  action  reguired. 

2-13.  CBTDEV  proponent  Drafts  COIC  for  MS  I  TEMP  (Figure  2-10) 

a.  CBTDEV  front  end  analysis.  The  process  begins  with  the 
CBTDEV  proponent  conducting  COIC  front-end  analysis.  This 
analysis  uses  as  its  base  the  MNS  approved  at  Milestone  0  and 
draws  from  ORD  formulation  process.  Additional  considerations 
include  when  available:  baseline  intelligence  products  (or  the 
System  Threat  Assessment  Report  (STAR));  system  critical 
mission(s)  and  function(s);  system  employment  and  sustainment 
concepts;  similar  system(s)  acquisition  experience; 
studies/analyses  used  to  justify  the  system  requirement;  the 
COEA;  and  recent  COIC  approval  process  experiences  (approved 
examples,  questions,  rejections,  and  guidance). 

NOTE:  Object  of  front-end  analysis  is  to  "get  system  and  COIC 

smart"  to  provide  properly  focused  COIC  with  appropriate  bottom 
line  critical  mission  accomplishment  and  sustainment  standards. 

b.  Draft  COIC  for  MS  I  TEMP.  With  submission  of  draft  ORD  to 
HQ  TRADOC  for  approval,  and  in  coordination  with  MATDEV  and  lOE, 
the  CBTDEV  proponent  prepares  draft  COIC  focused  on  the  full 
production  (MS  III)  decision.  The  critical  issues,  being  based 
on  the  MNS,  are  unlikely  to  change  as  the  system  proceeds  through 
the  acquisition  phases.  Criteria,  on  the  other  hand,  may 
Initially  be  "soft"  (i.e.,  lack  specificity).  Detailed 
pj^gparat ion  guidelines  are  found  in  Chapter  3,  this  Part. 
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***************insERT  figure  2-10************** 

2-14.  COIC  coordination  and  approval  for  MS  I  TEMP  (Figure  2-11) 

a.  CBTDEV  proponent  COIC  coordination.  Following  TRADOC 
approval  of  the  ORD  NLT  120  days  prior  to  MS  1/  the  CBTDEV 
proponent  makes  appropriate  adjustments  to  the  draft  COIC  and 
coordinates  them  with  the  MATDEV,  OPTEC,  Combined  Arms  Support 
Command  (CASCOM),  TRAC,  and  HQ  TRADOC.  Coordination  with  the 
MATDEV  will  ensure  synchronization  with  software  requirements 
definition  documentation,  RAM  (reliability,  availability,  and 
maintainability)  rationale  reporting,  and  system  specifications 
to  be  documented  in  a  request  for  proposal  (RFP).  Coordination 
with  the  operational  evaluator  will  ensure  that  COIC  are 
testable,  measurable,  or  otherwise  evaluatable  and  provide  an 
ppropriate  structure  for  the  operational  evaluation. 

Additionally  (Block  3A) ,  in  that  COIC  focus  on  the  full 
production  (MS  III)  decision,  the  CBTDEV  proponent  interfaces 
with  the  MATDEV  and  operational  evaluator  to  formulate  system 
performance  exit  criteria  for  the  MS  II  (Development)  decision 
review.  Coordination  with  TRAC  will  ensure  appropriate 
synchronization  with  the  COEA. 


NOTE:  A  discussion  of  the  relationship  of  exit  criteria  to  COIC 
can  be  found  in  Chapter  3,  this  Part. 


b.  CBTDEV  proponent  submits  COIC  for  approval.  The  CBTDEV 
proponent  forwards  the  coordinated,  final  draft  COIC  to  the 
TRADOC  center/school  (Proponent)  commander,  commandant,  or 
assistant  commandant  (as  appropriate)  for  review  and  approval. 

c.  COIC  approval  for  MS  I  TEMP.  The  CBTDEV  proponent 
(CDR/CMDT/AC)  approves  COIC  at  least  75  days  prior  to  MS  I  and 
distributes  them.  TRADOC  Regulation  71-3  provides  distribution 
guidance  for  TRADOC.  Minimum  distribution  includes  HQ  TRADOC 
(Block  5A),  the  operational  evaluator  (OPTEC  -  Block  5B)  for 
incorporation  in  the  TEP,  and  MATDEV  (Block  5C)  for  incorporation 
into  Part  IV,  Paragraph  4B  of  the  TEMP 

***************insert  figure  2-11************** 

2-15.  MS  I  and  Post-MS  I  COIC  activities  (Figure  2-12) 

a.  HQ  TRADOC  COIC  review.  HQ  TRADOC  applies  "management  by 
exception"  to  ACAT  III  and  IV  as  well  as  T/T  IMA  system  COIC.  If 
changes  are  directed  (Block  6A) ,  the  CBTDEV  proponent  resolves 
the  changes  and  their  impacts  (Block  6B),  re-coordinates  them 
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(Block  3),  and  resubmits  them  for  approval  (Blocks  4  and  5).  M 
approved  TEMP  is  also  required  for  MS  I  for  these  systemg_^ 

b.  OPTEC  evaluation  planning  for  MS  II.  OPTEC  incorporates 
the  approved  COIC  into  the  TEP  for  Early  Operational  Assessment 
(EOA)/  or  Abbreviated  Operational  Assessment  (AOA),  and  uses  them 
as  a  basis  for  finalizing  AOIC  (Chapter  3,  this  part)  to  guide 
the  MS  II  evaluation. 


c.  CBTDEV  identifies  required  COIC  changes.  The  operational 
requirement  matures  as  a  system  progresses  through  its 
acquisition  phases.  As  a  result,  revisions  to  COIC  may  1^® 
necessitated  to  ensure  that  they  continue  to  adequately  and 
accurately  represent  features/capabilities  critical  to  mission 
performance  and  provide  proper  focus  for  the  full  production 
decision.  After  approval  of  the  updated  ORD  and  COEA  for  MS  II, 
the  CBTDEV  proponent  normally  needs  to  restructure  criteria  to 
reflect  a  greater  level  of  specificity  (firmness)  than  that  found 
in  the  "soft"  criteria  of  the  initial  set  (i.e.,  the  criteria 
must  provide  an  operationally  meaningful  and  critical  measure  of 
mission  performance  for  which  a  shortfall  would  be  an  operational 
"show  stopper").  The  CBTDEV  proponent  identifies  necessity  tor 
change/update  by  actively  participating  in  and  maintaining  full 
cognizance  of  the  system's  developmental  progress.  First  among 
many  potential  sources  of  change  is  the  Acquisition  Decision 
Memorandum  (ADM)  or  "minutes"  of  the  In  Process  Review  (IPR) , 
which  documents  decisions  and  directives  and  approves  the  system 
concept  baseline  (MS  I)  and  exit  criteria  for  the  next  MS. 
Additional  sources  include,  but  are  not  limited  to,  results 
EUTE/EOA/AOA  (Block  8),  emerging  results  from  the  COEA,  the  ORD 
refinement/revision  process  (to  culminate  in  HQ  TRADOC  approval 
120  days  in  advance  of  MS  II)#  and  development  of  the  RAM 
Rationale  Report  and  materiel  specifications  for  the  RFP. 


**********INSERT  figure  2-12************** 

2-16.  COIC  update  and  approval  for  MS  II  TEMP  (Figure  2-13) 

a  CBTDEV  updates  COIC  for  MS  II  TEMP.  The  CBTDEV  proponent, 
in  conjunction  with  the  MATDEV,  SFTDEV  (theater/tactical  IMA 
systems),  and  operational  evaluator  (OEC),  updates  COIC  as 
necessary,  to  include  the  addition  of  "firm"  MS  III  criteria. 
Concurrently  (Block  lOA) ,  he  works  with  the  operational  evaluator 
and  MATDEV  in  the  formulation  of  LRIP  and/or  MS  III  performance 
oriented  exit  criteria. 


NOTE:  When  needed  performance  oriented  exit  criteria  for  MS  II 

were  formulated  pre-MS  I,  approved  during  MDR-I,  and  documented 
in  the  ADM.  In  the  event  of  a  consolidated  MS  I/II,  only  one  set 


2-13 


DRAFT  DA  PAM  73-1 
Part  Three 
14  OCT  92 


of  approved  documents  (including  COIC  and  TEMP)  will  be  produced, 
even  though  the  drafting  process  may  go  through  several 
iterations. 


b.  MS  II  TEMP  COIC  coordination.  The  CBTDEV  proponent 
coordinates  final  draft  COIC  with  the  MATDEV,  OPTEC,  CASCOM, 

TRAC,  and  the  TRASSO  not  later  than  115  days  in  advance  of  MS  II, 
submits  them  to  the  Proponent  for  review  and  approval . 
Concurrently,  the  MATDEV  Includes  the  final  draft  COIC  in  the 
TEMP  for  TIWG  staffing. 

c.  COIC  approval  for  MS  II  TEMP.  The  CBTDEV  proponent 
(CDR/CMDT/AC)  approves  for  TRADOC  the  COIC  for  ACAT  III  and  IV 
materiel  systems  as  well  as  theater/tactical  IMA  systems  not  on 
OSD  T&E  oversight.  Approval  and  distribution  must  occur  at  least 
75  days  in  advance  of  MS  II  for  on  schedule  approval  of  the  TEMP 
without  delay  of  MS  II.  CBTDEV  distributes  approved  COIC  to 
appropriate  acquisition  team  members.  TRADOC  Regulation  71-3 
provides  distribution  guidance  for  TRADOC.  The  following  are 
mandatory  recipients  of  COIC:  HQ  TRADOC  (Block  13A)  for 
information  and  action  deemed  appropriate,  the  operational 
evaluator  (OPTEC  -  Block  13  B)  for  Incorporation  into  the  TEP, 
and  the  MATDEV  (Block  13C)  for  incorporation  into  Part  IV, 
paragraph  4B  of  the  TEMP. 

******************INSERT  figure  2-13*************** 

2-17.  MS  II  and  Post-MS  II  COIC  operations  (Figure  2-14) 

a.  HQ  TRADOC  review.  HQ  TRADOC  applies  "management  by 
exception"  to  ACAT  III  and  IV,  as  well  as,  theater/tactical 
system  COIC.  If  changes  are  directed  (Block  14A),  the  CBTDEV 
resolves  the  changes  and  their  impacts  (Block  14B) , 
re-coordinates  them  as  in  Block  11,  and  resubmits  them  for 
approval  as  in  Blocks  12  and  13  (Block  14c). 

b.  OPTEC  evaluation  plannaing  for  MS  III.  OPTEC  incorporates 
approved  COIC  into  the  TEP  for  Initial  Operational  Test  and 
Evaluation  (lOTE)  and  uses  them  as  a  basis  for  finalizing  AOIC 
(Chapter  3,  this  part)  to  guide  the  MS  II  evaluation. 

c.  CBTDEV  identifies  required  COIC  updates. 

(1)  The  CBTDEV  proponent  continues  to  actively  participate 
in  and  maintain  full  cognizance  of  the  system's  developmental 
progress.  In  that  firm  MS  III  COIC  were  approved  and  included  in 
the  TEMP  at  MS  II,  COIC  will  not  be  further  modified  unless 
directly  affected  by  developmental  modifications  (i.e., 
preplanned  block  upgrades  and  other  modifications  to  the  system 
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responding  to  a  revised  ORD/MNS.  If  revision  is  necessitated, 
the  CBTDEV  proponent  reenters  the  process  and  completes  Blocks 
11-13,  in  support  of  the  full  production  decision.  For 
modifications,  the  full  production  decision  may  be  a  MS  III,  a 
decision  to  adopt  and  apply  an  engineering  change  proposal  (ECP), 
or  the  decision  to  procure  Modification  Work  Order  (MWO)  sets. 

(2)  For  system  developmental  modifications  after  initiation 
of  production/fielding,  a  TEMP  is  included  in  the  modification 
proposal  approval  package  which,  when  approved,  will  guide  the 
evaluation  and  decision  to  adopt  the  modification.  COIC  will  be 
an  integral  element  of  the  TEMP.  The  CBTDEV  therefore  reenters 
the  process  and  completes  Blocks  9-13  in  support  of  the 
modification  approval.  Paragraph  2-1,  this  chapter  identifies 
modifications  which  require  COIC. 

************INSERT  figure  2-14*********** 


COIC  Process  for  Strategic  and  Sustaining  Base  (S/SB)  Information 
Mission  Area  (IMA)  Systems 


2-18.  General  . 

The  development,  review,  and  approval  process  for  COIC  applicable 
to  strategic  and  sustaining  base  (S/SB)  information  mission  area 
(IMA)  systems  is  essentially  the  same  as  that  for  other  systems. 
Differences  are  predominantly  those  of  the 

organizations/activities  involved  and  timing  in  relation  to  the 
milestone  decision  review  (MDR) .  The  process  is  graphically 
depicted  in  Figure  2-15. 


***************insert  figure 


2-19.  Strategic  and  Sustaining  Base  (S/SB)  system  COIC 
development,  coordination,  and  approval  (Figure  2-15) 

a.  Functional  Proponent  (FP)  develops  COIC.  The  process 
begins,  at  least  140  days  prior  to  MDR  I  days,  with  the 
Functional  Proponent  (FP)  developing  COIC  based  on  the  MNS  and 
Functional  Description  (FD),  and  staffing  them  internally. 

b.  FP  forwards  COIC  to  HQDA  (ODISC4).  The  COIC  are  submitted 
to  ODISC4,  Analysis  and  Evaluation  Office  (HQDA,  SAIS-AE)  ,  at 
least  100  days  prior  to  MDR  I,  for  review  and  staffing.  This 
phase  also  Includes  a  COIC-MNS  relationship  briefing  to  the 
Chief,  Analysis  and  Evaluation  Office.  Section  V  of  this  chapter 
provides  format  guidance  for  the  briefing  package. 
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c.  COIC  approval  briefing  to  VDISC4.  A  decision  briefing  for 
COIC  approval  by  the  VDISC4  is  presented  at  least  93  days  prior 
to  MDR  I. 

d.  COIC  approval  and  distribution.  DISC4  approves  and  SAIS- 
AE  forwards  approved  COIC  to  PM/TIWG  chairperson  for  inclusion  in 
the  TEMP  at  least  79  days  prior  to  MS  I . 

e.  OPTEC  review  of  TEMP.  Following  OPTEC  review  of  the  TEMP, 
the  PM  fojcwards  it  to  TEMA  for  approval  at  least  65  days  prior  to 
MDR  I  . 


f.  TEMP  approval.  The  TEMP  is  approved  by  the  DUSA(OR)  at 
least  45  days  prior  to  MDR  I. 

g.  S/SB  IMA  system  COIC  update  or  revision.  As  for  other 
systems,  COIC  for  S/SB  IMA  systems  are  subject  to  revision  based 
on  changes  in  the  requirement  and  program  redirection  by  the  MDR 
decision  authority.  When  change  is  necessitated,  the  process 
flow  and  timing  for  subsequent  MDRs  or  other  events  (e.g., 
modification)  parallel  that  described  for  MDR  I. 


Section  V 

COIC-ORD  Crosswalk  Brief  Horse  Blanket 


2-20.  General 

The  COIC-ORD  Crosswalk  "Horse  Blanket"  serves  to  graphically 
depict  pertinent  system  information  and  the  direct  link  between 
the  ORD  requirement  and  COIC  criterion  in  support  of  the  DA 
(ADCSOPS-FD)  decision  (approval)  briefing  for  ACAT  I,  II,  OSD  T&E 
Oversight,  and  DA  selected  ACAT  III  and  IV  system  COIC.  The 
Horse  Blanket  (one  or  more  pages  measuring  approximately  30  X  38 
inches  each)  is  prepared  by  the  CBTDEV  in  the  format  depicted  in 
Figures  2-16  and  2-17. 

***************inseRT  figure  2-16**************** 

2-21.  Horse  Blanket  first  sheet  contents 

Page  one  (Figure  16)  will  contain,  as  a  minimum,  the  following 
Information: 

a.  A  concise  system  description.  Annotated  line  drawings  or 
schematics  may  be  used  as  appropriate  to  facilitate  this 
description. 

b.  The  Operational  Mode  Summary/Mission  Profile  (OMS/MP) . 
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c.  Threat  description.  A  brief,  unclassified  description  of 
the  key  threat (s)  to  the  system  that  will  exist  at  the  time  of 
fielding. 

d.  Operational  Scenario.  A  synopsis  of  the  operational 
scenario,  i.e.,  how  the  system  will  be  employed  on  the 
battlefield. 

e.  Program  status.  Program  status  to  include  funding  and 
schedule. 

***************INSERT  figure  2-17**************** 

2-22.  Horse  Blanket  sheets  2  through  n 

Pages  2  through  n  (Figure  2“17),  unless  fully  incorporated  on 
page  one,  will  contain  the  COIC-ORD  crosswalk.  All  COIC  (issue, 
scope,  criteria,  rationale,  and  notes)  will  be  presented  in  their 
entirety.  Applicable  ORD  requirement  and  rationale  paragraphs 
will  be  referenced,  stated  and  linked  by  color  coded  lines  to  the 
appropriate  criterion.  COIC  will  maintain  their  integrity  as 
submitted  for  approval.  ORD  requirements  will  be  cut, 
duplicated,  and  located  as  appropriate  and  necessary  to  support 
presentation  of  the  COIC  (i.e.,  ORD  requirements  section  will  not 
maintain  its  integrity) .  The  ORD  requirements  column  normally 
does  not  have  to  include  all  ORD  operational  will  not  include  all 
does  not  requirements. 

2-23.  COIC  approval  pre-brief 

Under  DA  ADCSOPS-FD  Operational  Requirements  Directorate 
(DAMO-FDR)  lead,  the  COIC-ORD  crosswalk  pre-brief  is  two  to  three 
days  in  advance,  normally  at  the  ADCSOPS-FD  Integration 
Directorate  (General  Officer)  level,  to  determine  readiness  for 
delivery  of  the  decision  briefing  to  the  ADCSOPS-FD.  Attendees 
include  representatives  from  the  PM  office,  DAMO-FDR,  OEC,  HQ^ 
TRADOC,  and  the  TRADOC  proponent  center/school  and/or  TSM  office. 

2-24.  COIC  approval  brief 

a.  The  ADCSOPS-FD  decision  briefing,  for  approval  of  the 
COIC,  occurs  not  later  than  100  days  in  advance  of  Milestone  II. 
The  principal  briefer  is  a  representative  of  the  TRADOC  proponent 
school  Director  of  Combat  Developments  (DCD)  or  the  TRADOC  System 
Manager  (TSM).  Required  attendees  (in  addition  to  the  briefer 
and  DAMO-FDT  representative)  include  the  DA  DCSOPS  System 
Integrator  (SI),  the  HQ  TRADOC  System  Staff  Officer  (TRASSO) ,  the 
TRADOC  proponent  school  DCD  or  the  TSM,  the  Program  Manager  (PM), 
a  representative  from  OPTEC  OEC,  and  a  representative  from  DA 
DCSINT.  Others  (subject  to  space  available)  may  attend  as  needed 
to  answer  specific  questions. 
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b.  Any  thing  on  the  Horse  Blanket  is  subject  top  challenge 
but  particularly  the  OMS/MP,  ORD  requirements/  ORD  requirements 
rationale,  and  the  COIC.  Examples  of  questions/areas  of  concern 
surfaced  during  the  decision  briefing  include: 

(1)  Would  you  withhold  program  go-ahead  if  this  criterion 
is  not  achieved?  NOTE;  If  the  criterion  was  properly  selected 
and  structured  as  a  "show  stopper,"  the  answer  should  be  "yes." 

A  "No"  response  will  cause  the  ADCSOPS-FD  to  direct  rework  of  the 
COIC. 


(2)  What  is  the  critical  mission  to  be  performed?  NOTE: 
This  question  is  avoidable  in  that  system  COIC  should  include  one 
or  more  mission  performance  Critical  Operational  Issues  (COI). 

(3)  How  will  the  system  be  supported?  NOTE:  Because  there 
is  no  mandatory  requirement  for  a  sustainment  issue,  the  subject 
of  this  question  may  not  be  addressed  by  the  system  COIC. 
Attendees  should,  however,  be  prepared  to  answer  this  type 
question  as  an  element  of  the  system  operational  scenario. 

(4)  What  is  the  basis  for  this  ORD  requirement?  NOTE: 

This  question  should  be  avoidable  if  the  "Horse  Blanket"  is 
properly  structured  to  include  the  requirement  rationale  from  the 
ORD. 


(5)  When  was  the  OMS/MP  last  updated  and  on  what  basis? 
NOTE:  The  OMS/MP,  as  an  element  of  the  RAM  Rationale  Report 
(RRR)  supporting  the  ORD,  should  have  been  updated  by  the  CBTDEV 
following  Milestone  I,  approved  by  HQ  TRADOC  NLT  230  days  prior 
to  Milestone  II,  and  approved  by  HQDA  NLT  170  days  prior  to 
Milestone  II. 

(6)  How  will  this  criteria  be  evaluated/tested?  NOTE: 

This  question  is  partially  answerable  from  the  "Horse  Blanket"  if 
the  COIC  (including  scope)  are  included  in  their  entirety  as 
required.  The  Operational  Evaluator  must  be  prepared  to  provide 
detailed  answer. 

(7)  What  does  this  term  mean  operationally?  Attendees  must 
be  familiar  with  the  definition  and  operational  Impact  of  all 
terms  used  in  the  COIC.  If  not,  they  can  jeopardize  the 
successful  outcome  of  the  briefing. 

2-25.  COIC  "Horse  Blanket"  for  Information  Mission  Area  (IMA) 
systems 

The  IMA  system  COIC  "Horse  Blanket"  is  similar  to  that  for 
materiel  systems.  The  "Horse  Blanket"  is  used  to  brief  ADCSOPS- 
FD  for  theater/tactical  IMA  systems  and  DISC4  for  strategic  and 
sustaining  base  IMA  systems.  The  following  categories  apply: 
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a.  System  Description. 

b.  Operational  Mode  Summary/Mission  Profile. 

c.  Threat  statement. 

d.  System  Deficiencies. 

e.  Functional  Improvements. 

f .  Programatics . 

(1)  Acquisition  Plan/Program  Schedule. 

(2)  Funding. 

g.  MNS/FD  -  COIC  Crosswalk. 

(1)  MNS/FD  Requirement. 

(2)  COIC. 
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Chapter  3  ^  j  „ 

COIC  Development  Considerations  and  Guidelines 

Section  I 
General 


3-1.  COIC  purpose 

a.  Primary  purpose.  The  primary  purpose  of  COIC  is  to  focus 

and  support  milestone  decisions.  They  prescribe  (and  provide  a 
consistent  emphasis  on)  the  user’s  minimum  operational 
effectiveness  and  suitability  expectations  from  the  . 

operational  system  for  a  go-ahead  decision  at  the  full  production 
(normally  MS  III)  decision.  COIC  reduce  the  multitude  of 
operatioLl  considerations  to  a  ^ 

relevant,  mission  focused  issues  and  criteria.  The  COIC  ar 
relevant  to  both  the  combat  mission  operations  and  the  full 
production  decision,  integrating  operational  mandates  with 
maturity  considerations  for  the  total  operational  system.  COIC 
are  initially  developed  for  the  TEMP  approved  prior  to  MS  I  and 
updated  as  necessary  thereafter.  For  intermediate  milestone 
decisions  (e.g.,  MS  II  (Development  Approval)  and  LRIP 
authorization),  the  COIC  provide  operational  focus  and  essential 
objectives  to  assist  in  judging  the  operational  azimuth  and 
potential  of  the  concept  or  prototype. 

b.  Secondary  purposes.  Secondary  purposes  of  COIC  include 
serving:  to  focus  and  prioritize  the  operational  evaluation 
effort  (e.g.,  the  operational  evaluator  reports  system  status 
against  COIC  for  a  decision  review);  to  identify  operational 
priorities  for  the  acquisition  effort  (e.g.,  the  system  mus 
satisfy  criteria  or  be  otherwise  operationally  justified  to 
proceed);  and  to  foster  a  coordinated  effort  by  the  members  of 
the  acquisition  team  by  identifying  what  is  operationally 
important  (e.g.,  provide  operational  emphasis  and  focus  for 
CBTDEV  proponent  and  OIE  assistance  in  MATDEV  formulation  of 
milestone  decision  review  performance  exit  criteria) . 

c.  COIC  relationship  to  operational  test.  COIC  ^te  not 
Operational  Test  (OT)  issues  and  criteria.  As 

primary  purpose  of  COIC  is  to  support  milestone  decisions  and  , 

secondarily,  the  operational  evaluation.  TeJt 

can  come  from  any  credible  source  (e.g..  Initial  Operational  Test 
(lOT),  other  operational  test  (OT),  developmental 

field  data  collection,  studies/simulations,  etc.).  While  Initial 
Operational  Test  (lOT)  is  mandatory  by  law  for  ACAT  I  programs, 
other  programs  may  not  require  OT  for  the  full 
III)  decision.  It  is  therefore  necessary 

operational  evaluator  determine,  and  document  in  ^he  Temp,  the 
appropriate  data  source  for  COI  resolution.  The  CBTDEV 
proponent/FP  view  point  during  development  and  approval  of  COIC 


3-1 


DRAFT  DA  PAM  73-1 
Part  Three 
14  OCT  92 

must  be,  "This  is  what  is  needed  to  make  the  full  production 
decision,"  regardless  of  how  the  issue  will  be  answered.  As  long 
as  the  COIC  are  "musts"  for  the  decision,  and  can  be  answered  by 
the  independent  operational  evaluator,  they  are  good. 

3-2.  COIC  concept 

COIC  are,  by  definition,  those  decision  maker  key  operational 
concerns  (issues)  with  bottom  line  standards  of  performance 
(criteria)  which,  if  satisfied,  signify  that  a  system  is 
operationally  ready  to  proceed  during  the  full  production  (MS 
III)  acquisition  decision  review. 

a.  Critical  operational  Issues.  Key  operational  concerns 
(critical  operational  issues)  are  those  which  must  be  answered 
for  the  full  production  (MS  III)  decision  to  proceed.  They  are 
operationally  oriented  and  not  technology,  cost,  or  politics 
focused.  A  system  is  considered  operationally  ready  (effective 
and  suitable  enough)  to  proceed  to  full  production  when  the 
following  operational  concerns  are  answered  affirmatively 
(critical  operational  issue  categories): 

(1)  Can  the  system  accomplish  its  critical  mission(s)? 

(2)  Can  the  system  maintain  preparedness  for  and  be 
sustained  during  combat? 

Criteria.  Bottom  line  standards  of  performance  (the 
criteria)  are  show-stoppers  if  not  satisfied  for  the  full 
production  (MS  III)  decision.  If  a  shortfall  exists  for  one  or 
more  of  the  COIC  criteria,  convincing  evidence  (i.e.,  revised 
effectiveness,  sustainability,  and  cost  analyses  and  resulting 
considerations  and  review  of  program  alternatives)  must  be 
provided  for  the  decision  authority  to  allow  the  program  to 
proceed.  Like  the  issues,  the  criteria  are  operationally 
oriented  and  not  technology,  cost,  or  politically  focused.  This 
does  not  mean  that  the  criteria  are  operational  test  oriented, 
just  that  the  criteria  provide  operationally  relevant  measures. 
While  most  criteria  will  be  answered  using  data  from  some  form  of 
operational  test,  some,  such  as  NBC  contamination  hardening  (when 
a  specific  program  objective),  must  depend  on  technical  test  or 
simulation  data. 

c.  Total  operational  system  focus.  The  system  of  concern  is 
the  total  operational  system  (See  Figure  3-1)  as  a  composite 
rather  than  any  of  its  component  parts.  Simultaneously,  the 
total  system  of  Interest  may  be  a  single  system  (e.g.,  truck  with 
trailer)  or  an  operational  unit  (e.g.,  a  team  or  platoon).  This 
has  several  benefits,  not  the  least  of  which  is  fewer  issues. 

Other  key  benefits  are  that  they  are  more  relevant  to  operations 
than  if  focused  on  system  components,  and  the  potential  for 
duplicate  coverage  is  reduced. 
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***********INSERT  figure  3-1**************** 

d.  COIC  structure.  COIC  format  provides  for  each  issue:  a 
scope  paragraph  (conditions  for  evaluating  the  issue) ,  its 
associated  criteria,  and  a  rationale  section  (justification  for 
each  criteria).  Additionally,  the  structure  provides  a  notes 
section  including  two  standardized  mandatory  notes  (the  first 
addressing  the  total  system  focus  and  coverage  of  the  criteria; 
the  second  addressing  the  pass/fail  application  of  the  CO  C)  an 
other  system  specific  notes  as  needed.  A  third  mandatory  ^ote 
(stating  that  COIC  are  based  on  initial  requirements  and  will  be 
updated  prior  to  MS  II)  is  included  for  COIC  supporting  the  MS  I 
TEMP.  As  the  structure  indicates,  the  criteria  are  the 
instruments  for  judging  whether  an  issue  is  satisfied 
(achievement  of  all  criteria  results  in  a  satisfied  issue).  This 
structure  applies  to  COIC  coordination,  approval,  and  processing, 
TEMP  content,  and  TEP  content.  COIC  are  coordinated,  staffed, 
and  approved  as  a  stand  alone  document.  Figure  3-2  provides  more 
details  on  the  COIC  coordination  and  submission  format. 

e  Initial  COIC  development  and  update.  COIC  are  Initially 
developed,  approved,  and  included  in  the  TEMP  approved  prior  to 
MS  I .  They  are  subsequently  updated  as  needed  as  the  program 
progresses  (particularly  in  response  to  the  ORD  TEMP  updates 

prior  to  MS  II).  The  issues,  being  based  on  the  MNS,  will  seldom 
change;  however,  the  criteria  will  change  as  the  operational 
requirement  matures  and  in  response  to  significant  program 
restructures  (e.g.,  shifting  of  "block"  improvements).  Criteria 
for  the  COIC  applicable  to  the  MS  I  TEMP  (even  if  updated  as  a 
result  of  or  in  conjunction  with  a  restructure  of  the 
Demonstration  and  Validation  Phase  effort)  may  be  soft  (provide  a 
performance  standard  but  not  a  final  performance  threshold). 
Criteria  for  the  COIC  applicable  to  the  MS  II  TEMP  will  be  firm, 
measurable  performance  thresholds. 

***************INSERT  FIGURE  3-2**************** 


3-3.  Front  end  analysis 

a.  Key  system  knowledge  to  attain.  As  with  many  processes, 
that  for  COIC  first  requires  that  the  writers  do  the  necessary 
research  -  lay  the  groundwork  -  which  will  serve  as  a  foundation 
for  his  efforts.  In  short,  he  must  "get  smart"  on  his  system  and 
lessons  learned  on  similar  systems  to  do  a  credible  job.  Key 
considerations  include,  but  are  not  necessarily  limited  to: 

(1)  The  reason(s)  for  a  new  system  or  modification  to  an 
existing  system  -  the  need. 

(2)  The  system's  critical  mission(s)  and  function(s). 
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(3)  The  system's  employment  and  sustainment  concepts. 

(4)  The  system's  acquisition  status. 

(5)  Acquisition  experiences  for  similar  systems,  e.g., 
reasons  for  T&E  or  proceed  decision  problems,  etc. 

(6)  Recent  COIC  approval  process  experiences,  e.g., 
examples  of  and/or  lessons  learned  from  those  which  engendered 
questions,  guidance,  rejection,  or  approval  by  decision  makers. 

b.  Key  information  sources.  Key  documents  which  serve  as 
sources  for  "get  smart"  information  are  itemized  below.  Each  is 
important  in  its  own  right,  but  of  greater  importance  is  their 
contribution  to  the  synergism  of  system  documentation.  It 
therefore  behooves  the  COIC  writer  to  become  intimately  familiar 
with  their  purpose  and  content  before  he  proceeds.  This  list  is 
not  meant  to  be  exhaustive;  therefore,  the  COIC  writer  must 
during  this  phase  be  careful  to  identify  those  other  system 
peculiar  documents  with  information  in  the  above  areas.  It 
should  be  noted  that  system  requirements  documented  before 
February  1991  (publication  of  DoDD  5000.1,  DoDI  5000.2,  and  DoD 
5000. 2-M)  made  use  of  the  Operational  and  Organizational  (O&O) 
Plan  and  Required  Operational  Capability  (ROC),  while  those  after 
Feb  91  made  use  of  the  MNS  and  ORD. 

(1)  Mission  Need  Statement  (MNS). 

(2)  Operational  Requirements  Document  (ORD). 

(3)  Software  Functional  Description  (FD) . 

(4)  System  Threat  Assessment  Report  (STAR). 

(5)  Cost  and  Operational  Effectiveness  Analysis  (COEA) . 

(6)  Reliability,  Availability,  and  Maintainability  (RAM) 
Rationale  Report  (RRR). 

(7)  Milestone  Decision  Review  (MDR)  Minutes. 

(8)  System  Specifications  (Request  for  Proposal  -  RFP). 

3-4.  Relationships 

In  the  broadest  sense,  COIC  are  derived  from  the  documented 
operational  requirement  to  reflect  those  minimum  essential 
operational  concerns  and  operational  performance  standards  key 
for  achievement  of  full  production  authorization  at  the  MS  III 
decision  and  to  serve  as  the  priority  focus  for  the  supporting 

evaluation.  In  detail,  these  Inherent  relationships 
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to  the  requirement/  the  decision  process,  and  the  supporting 
evaluation  are  more  complex. 

***************INSERT  figure  3-3******************** 


a.  COIC  and  the  operational  requirement.  The  operational 
requirement,  along  with  key  employment  considerations,  are 
essential  to  establishing  operationally  valid,  relevant,  and 
credible  COIC.  As  depicted  in  Figure  3-3,  the  operational 
requirement  presents  itself  in  many  forms  dependent  on  the 
system.  If  the  COIC  developer  and  the  Acquisition  Team  do  their 
jobs  correctly,  there  will  be  compatibility  between  COIC  and  the 
key  documents  listed. 


(1)  COIC  and  the  operational  requirements  documents 
(MNS/ORD) .  For  materiel  system  acquisitions,  the  critical 
operational  issues  will  be  based  on  the  MNS  and  thus  unlikely  to 
change  as  the  program  proceeds.  The  criteria  will  be  based  on 
the  ORD  and  thus  change  as  the  requirement  matures .  Automated 
information  systems  which  are  not  part  of  a  materiel  acquisition 
do  not  have  a  document  comparable  to  the  ORD;  therefore,  both 
their  issues  and  criteria  are  based  on  the  MNS.  The 
for  AIS  uses  the  Functional  Description  (FD)  to  support  the  MNS. 
(A  note  of  caution  applies  here  regarding  the  potential  for 
technical  measures  and  lack  of  operational  relevance  for  some 
ROC/ORD/FD  requirements.  The  COIC  writer  is  often  better  served 
by  the  rationale  for  the  requirement  than  by  using  the  actual 
requirements.)  "Being  based  on"  does  not  mean  that  issues  and 
criteria  are  direct  lifts  from  these  documents,  but  that  there  is 
a  clear,  auditable  foundation  for  the  issues  and  criteria  in 
these  documents.  For  example,  the  ORD  may  require  a  significant 
survivability  improvement  over  the  existing  system,  whereas  the 
COEA  and  cost  considerations  may  result  in  a  criteria  to  complete 
20  percent  more  missions  with  50  percent  more  threat  neutralized. 
The  COIC  rationale  provide  a  crosswalk  between  the  ORD  minimum 
acceptable  requirements  and  the  COIC. 


(2)  COIC  and  the  Cost  and  Operational  Effectiveness 
Analysis  (COEA) .  The  COEA  is  the  primary  analytical  document  of 
operational  consideration  during  MS  I  and  MS  II  decisions.  It 
compares  the  relative  cost  and  operational  effectiveness  for 
ive  concepts  considered  and  indicates  their  relative 
status  to  the  baseline.  As  such,  it  has  significant  influence  on 
the  concept  chosen  to  proceed.  This  influence  has  a  significant 
bearing  on  the  requirement  document  and  COIC.  The  COEA  provides 
relevant  effectiveness  and  cost  considerations  such  as 
significantly  improved  performance  at  significantly  reduced  cost. 
For  instance,  if  the  COEA  shows  a  significant  cost  saving  over 
the  baseline  and  this  is  the  purpose  of  the  acquisition 
(modernization),  then  the  criteria  should  reflect  a  system  which 
is  as  mission  capable,  trainable,  and  sustainable  in  combat  as 
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the  existing  system.  The  COEA  uses  various  measures  of 
performance  (MOP)  for  which  sensitivity  runs  could  aid  in 
establishing  criteria  for  the  COIC.  Because  of  the  significance 
of  the  COEA  to  the  program,  there  must  be  an  audit  trail  of 
consideration  between  the  COIC  and  the  COEA. 

(3)  COIC  and  the  system  specification.  The  primary  concern 
here  is  compatibility  between  the  COIC  and  the  specification  (or 
contract  represented  by  the  specification).  If  COIC  are  fully 
documented  in  the  ORD,  there  is  no  reason  for  incompatibility 
between  the  specification  and  the  ORD  since  the  specification 
must  be  compatible  with  the  ORD.  The  MATDEV  assures  this 
compatibility,  or,  when  COIC  are  developed  late,  advises  when  an 
incompatibility  exists.  Occasionally  the  specification  serves  to 
document  operational  performance  parameters  which  did  not  make  it 
into  the  ROC/ORD  (e.g.,  the  ROC  requires  a  level  of  connectivity 
for  communications  users  -  the  specification  requires  a  level  of 
successful  transmission  given  connectivity  exists). 

(4)  COIC  and  other  requirements  documents  (Studies  and 
Cost).  When  neither  the  MNS/ROC/ORD/FD,  COEA,  nor  specification 
provide  all  requirement  information  needed  to  develop  valid  COIC, 
then  other  sources  are  tapped.  Most  of  the  time  these  aspects 
are  considered  in  establishing  MNS/ROC/ORD/FD  requirements  (e.g., 
operation  and  support  costs  are  used  in  establishing  RAM 
requirements  considered  during  COIC  development). 

b.  COIC  and  operational  employment  considerations.  To 
produce  operationally  realistic  and  valid  COIC,  the  COIC  writer 
must  understand  and  continually  focus  on  the  operational 
mission(s)  (described  in  the  Operational  Mode  Summary/Mission 
Profile  (OMS/MP)  appendix  to  the  .ORD)  and  system  employment 
tactics,  techniques,  and  procedures.  An  understanding  of  how  the 
system  fights/operates  is  critical  to  determining  if  system  or 
organizational  type  measures  should  apply  (e.g.,  a  system  which 
fights  as  an  element  of  a  platoon  of  ten,  with  target  detection 
and  hand-off  for  engagement  accomplished  Internal  to  the  platoon, 
should  not  be  measured  as  a  single,  stand  alone  system). 
Similarly,  an  understanding  of  how  system  operations  will  be 
logistically  supported  is  essential  in  defining  sustainment  COIC. 
Operational  requirements  must  therefore  be  examined  in  light  of 
operational  employment  considerations  to  arrive  at  meaningful 
criteria  for  COIC. 

c.  COIC  and  performance  exit  criteria.  COIC  criteria,  by 
definition,  are  bottom  line  standards  which,  if  satisfied, 
indicate  that  a  system  is  operationally  ready  to  proceed  to  full 
production.  Exit  criteria,  meanwhile,  are  established  in 
accordance  with  DoDI  5000.2  at  each  milestone  for  the  next 
milestone  (e.g.,  at  MS  II  for  MS  III)  and  for  major  events 
between  milestones  (e.g.,  long  lead  time  procurement  and  LRIP 


3-6 


DRAFT  DA  PAM  73-1 
Part  Three 
14  OCT  92 

authorizations).  They  are  minimum  requirements  that  must  be 
successfully  demonstrated  for  the  program  to  proceed. 

Performance  exit  criteria,  as  such,  serve  as  decision  point 
measures  of  progress  ("stepping  stones")  toward  achievement  of 
COIC  criteria  and,  eventually,  mature  system  objective 
performance.  While  the  CBTDEV  proponent  has  the  lead  in 
developing  the  full  production  (MS  III)  COIC,  the  MATDEV  has  the 
lead  in  developing  exit  criteria  and  does  so  with  the  assistance 
of  the  CBTDEV  proponent  and  in  coordination  with  the  independent 
operational  evaluator.  Most  MS  II  performance  exit  criteria  will 
measure  feasibility  of  fulfilling  operational  needs.  The  full 
production  performance  exit  criteria  and  COIC  focus  on  a  mission 
capable,  affordable  system.  The  relationship  of  COIC  and 
performance  exit  criteria  is  depicted  in  Figure  3-4.  This  figure 
represents  a  criteria  compendium  as  the  system  moves  from  MS  II 
to  MS  III.  Section  IV,  this  chapter  contains  a  detailed 
discussion  of  criteria  for  critical  operational  issues. 


***********insERT  figure  3-4************** 

d.  COIC  and  the  operational  evaluation.  The  Operational 
Independent  Evaluator  (OIE)  is  responsible  for  planning  an 
operational  evaluation  that  will  answer  the  COIC  for  the  full 
production  (MS  III)  decision.  Any  source  of  data  (e.g., 
operational  test,  developmental  test,  study,  and/or  survey) 
judged  credible  by  the  OIE  can  be  used  to  answer  the  COIC.  The 
evaluator  reports  the  system  achievement  against  the  COIC  at  the 
full  production  decision.  Plans  and  reports  for  follow-on 
operational  evaluations  will  use  these  same  COIC.  The  COIC  are 
first  documented  in  the  TEMP  prior  to  Milestone  I  to  influence 
the  program  and  operational  evaluation  planning  and  conduct 
leading  to  Milestone  II.  Additional  responsibilities  for  the  OIE 
include: 

(1)  Providing  an  evaluation  of  the  operational  status  of 
the  system  and  readiness  to  proceed  at  Milestone  II  and^ 
subsequent  milestone  decisions.  When  tasked  by  the  decision 
authority,  providing  follow— on  operational  evaluations  after 
Milestone  III  to  address  correction  of  shortcomings  found  at 
Milestone  III. 

(2)  Providing  a  determination  whether  the  minimum 
acceptable  operational  performance  requirements  stated  in  the  ORD 
have  been  satisfied. 

(3)  Providing  a  complete  and  comprehensive  evaluation  of 
the  system's  operational  effectiveness  and  suitability.  This 
includes  being  able  to  isolate  (or  point  in  the  direction  of)  the 
cause  of  operational  shortfalls  whenever  possible. 
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(4)  Identifying  the  data  required  from  operational  test, 
technical  test,  studies,  and  other  sources  to  accomplish  the 
evaluation  (i.e.,  define  the  specific  test,  study,  and  other 
issues  necessary  for  evaluation) . 

(5)  Providing  a  baseline  comparison  assessment  for  the  full 
production  decision. 

To  accomplish  these  many  responsibilities,  the  OIE  prepares  a 
more  definitive  set  of  operational  issues  and  criteria  known  as 
Additional ^Operational  Issues  and  Criteria  (AOIC).  The  OIE 
documents  the  AOIC  in  Chapter  2  of  the  operational  Test  and 
Evaluation  Plan  (TEP). 

***************inseRT  FIGURE  3-5*************** 
e.  COIC  and  AOIC. 

(1)  While  the  focus  of  COIC  is  on  the  minimum  needed  to 
know  what  is  good  enough  operationally  for  a  go-ahead  decision  at 
the  full  production  decision  point,  AOIC  focus  on  a  complete  and 
comprehensive  evaluation  of  the  system's  operational 
effectiveness  and  suitability.  COIC  concern  is  an  operationally 
effective  and  suitable  total  system,  as  evidenced  by  the  total 
system's  readiness  for  and  capability  to  sustain  accomplishment 
of  critical  missions  during  combat.  AOIC  concern  is  for  the 
operational  effectiveness  and  suitability  of  the  total  system  as 
evidenced  by  performance  of  the  components  of  operational 
effectiveness  and  suitability  (see  Figure  3-5). 

(2)  AOIC  are  developed  by  the  OIE  based  on  the  COIC,  MNS, 
ROC/ORD,  system  specification,  COEA,  applicable  regulations  and 
pamphlets,  RAM  Rationale  Report,  and  other  sources.  AOIC  are 
structured  in  the  same  basic  four-part  format  as  COIC  (i.e., 
issue  with  associated  scope,  criteria,  and  rationale).  Since 
AOIC  support  a  complete  and  comprehensive  evaluation,  they  tend 
to  be  more  diagnostic  and  may  include  investigative  issues  (start 
with  "How  well"  or  "What  is")  which  do  not  require  criteria. 

COIC  never  include  investigative  issues.  AOIC  support  COIC 
resolution  as  follows  (see  Figure  3-6): 

*******************INSERT  FIGURE  3-6******************* 

(a)  Allow  the  OIE  to  specify  the  data  required  from 
multiple  sources  (in  the  form  of  issues  and  criteria)  for  COIC 
not  directly  answerable  from  operational  test.  For  tester, 
analyst,  and  evaluator  execution  purposes,  these  AOIC  are  just  as 
critical  as  the  COIC  they  support.  If  the  data  are  not  provided, 
the  OIE  will  not  be  able  to  evaluate  the  issue  for  the  full 
production  decision  -  thus,  no  decision. 
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(b)  Provide  the  OIE  the  diagnostics  to  identify  factors 
contributing  to  or  causing  a  performance  shortfall  for  one  or 
more  of  the  COIC. 

(c)  Complement  the  COIC  by  providing  a  comprehensive 
evaluation  of  all  aspects  of  the  total  operational  system.  In 
the  event  of  a  performance  shortfall  for  one  or  more  COIC,  the 
AOIC  may  provide  the  evidence  needed  to  convince  decision  makers 
that  the  system  is  good  enough  to  proceed  (e.g.,  baseline 
comparison  or  minimum  acceptable  operational  requirements 
accomplishments).  In  the  event  the  COIC  are  satisfied,  the  AOIC 
may  identify  areas  for  continued  improvement  as  the  system 
proceeds  in  acquisition. 

Section  II 

Identifying  the  Issues 
3-5.  Characteristics 

Critical  operational  issues,  by  definition,  are  those  key 
operational  concerns,  expressed  as  questions,  which,  when 
answered  completely  and  affirmatively,  signify  that  a  system  or 
materiel  change  is  operationally  ready  to  transition  to  full 
production.  They  are  few  in  number,  based  on  the  MNS,  and 
focused  on  the  full  production  (MS  III)  decision.  There  are  four 
key  components  of  a  properly  structured  critical  operational 
issue  statement: 

a.  The  interrogative.  An  interrogative  word  demanding  a 
"yes"  or  "no"  answer  (e.g..  Does,  Can,  or  Is). 

b.  The  system.  Identification  of  the  system  of  concern 
(e.g.,  system  X  or  a  platoon  equipped  with  system  X). 

c.  The  capability.  A  capability  of  concern  (e.g.,  robust 
voice  and  data  communication  or  effective  aerial  reconnaissance) . 

d.  The  conditions.  The  set  of  applicable .operational 
conditions  (e.g.,  during  combat  operations  or  as  employed  by 
Special  Operations  Forces). 

3-6.  Focus 

a.  Total  operational  system  concern.  Critical  Operational 
Issues  (COI)  focus  on  the  total  operational  system  as  an  entity 
and  its  ability  to  satisfy  the  operational  deficiency  or 
efficiency  defined  in  the  Mission  Need  Statement  (MNS).  This 
overarching  focus  for  COIC  results  in  a  few  issues  which  seldom 
change  as  the  system  progresses  through  the  acquisition  process. 
While  the  norm  is  three  issues  (e.g.,  one  mission  capability,  one 
deployability/mobility/interoperability,  and  one  sustainability), 
as  few  as  one  (single  shot  item  or  materiel  change)  or  as  many  as 
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six  (a  family  of  trucks)  may  be  appropriate.  This  focus  breaks 
the  mind  set  of  separate  operational  effectiveness  and 
suitability  issues.  A  single  issue  will  often  cover  the  areas  of 
mission  performance,  survivability,  RAM,  MANPRINT,  and  software 
performance  (e.g.,  probability  of  successful  communications  for  a 
communications  net  or  probability  of  kill  for  a  direct  fire 
weapon) . 

b.  COI  relevancy.  Operational  relevancy  translates  as 
"accomplish  critical  mission(s)"  and  "prepared  for  and  sustained 
in  combat."  "Accomplish  critical  mission(s)"  means  not  only  that 
the  system  is  capable  of  performing  its  mission  functions,  but 
that  it  survives  to  the  degree  needed  during  the  mission,  can 
interoperate  with  Army,  Allied,  and  Other-Service  systems 
necessary  for  mission  success,  and,  for  rapid  deployment  and 
remote  units,  is  deployable  to  the  site  of  combat  operations. 
NOTE:  For  other  than  rapid  deployment  or  remote  units, 
deployability  may  be  a  sustainability  issue.  "Prepared  for  and 
sustained  in  combat"  means  that  crews  must  maintain  proficiency 
in  garrison  and  planned  logistics  support  must  provide  responsive 
maintenance,  supply,  and  transportation  for  the  system  during 
combat . 

c.  COI  development  procedure.  From  the  view  of  minimizing  the 
COI,  preparation  of  the  COIC  starts  with  the  mission 
accomplishment  issue.  Normally  a  good  procedure  to  stating  the 
critical  mission  accomplishment  issue  is  to  frame  critical 
mission/task  order  to  be  given  by  higher  headquarters  as  the 
issue  (e.g..  Can  the  unit  equipped  with  system  x  take  and  hold 
the  tactical  objective  on  the  future  battlefield?  or  Can  truck  x 
pick  up  and  transport  required  tactical  loads  to  objective 
location  as  required  in  support  of  combat  operations?).  Complete 
that  issue  with  its  scope,  criteria,  and  rationale.  Then,  if 
there  is  anything  remaining  unaddressed  in  the  mission 
accomplishment  area,  define  that  issue  with  its  scope,  criteria, 
and  rationale,  remaining  cognizant  of  the  first  issue  and 
criteria  to  avoids  duplication  or  overlapping  coverage.  Once  the 
mission  area  is  complete,  consider  the  need  for  a  sustainment 
issue.  If  considered  not  needed,  provide  rationale  in  your  cover 
memorandum  when  coordinating  the  COIC  and  when  submitting  the 
COIC  for  approval.  Once  the  set  of  COIC  is  complete,  review  it 
for  duplication  or  overlapping  coverage,  and  eliminate  any  issue 
subsumed  by  another. 

3-7.  Developing  the  issue  -  Questions  to  Ask 

a.  What  is  the  system  of  interest?  For  example:  individual 
system  (tank,  rifle,  etc),  system  of  systems  (communications 
network),  or  system  component  change  (improved  missile  warhead). 
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b.  Why  the  system  (or  materiel  change)?  For  example:  the 
deficiency  the  system  is  being  designed  to  correct  or  opportunity 
it  is  intended  to  seize. 

c.  What  is  (are)  the  critical  mission(s)?  For  example: 
consider  all  against  the  question,  "Which  mission  requirement(s) , 
if  not  satisfied,  will  engender  a  "No-Buy"  decision?",  and,  where 
there  are  more  than  one,  "Which  mission  is  the  more 
rigorous/demanding? " . 

d.  Are  there  critical  user  unit  concerns?  For  example,  is 
the  system  deployable  by  light  forces  -  if  not,  is  a  "No-Buy" 
decision  in  order? 

e.  What  are  concerns  regarding  sustainment?  For  example,  is 
Ammunition  Supply  Point  (ASP)  throughput  capacity  sufficient  to 
support  a  significantly  higher  rate  of  fire  capability  for  a 
cannon  artillery  system? 

3-8.  Developing  the  issue  -  DOs  and  DON'Ts 

NOTE:  Each  DO  is  followed,  when  appropriate,  by  one  or  more 
companion  DON'Ts. 

a.  Focus.  DO  focus  the  issue  so  as  to  focus  the  evaluation 
and  decision.  State  a  question  which  aslcs  if  a  task  can  be 
performed  under  the  conditions  of  concern  (e.g..  Does  the  Nipper 
effectively  close  with,  detect,  engage,  and  destroy  threat  armor 
under  expected  battlefield  conditions?). 

DON'T  over  generalize  (e.g..  Is  the  Nipper  operationally 
effective?  or  "Is  the  Nipper  operationally  suitable?). 

DON'T  include  criteria  in  the  issue  statement  (e.g..  Does 
the  Nipper  find  and  kill  "X"  percent  of  threat  armor  within  its 
area  of  operations?). 

b.  Decision  issue.  DO  formulate  the  issue  as  a  question 
which  demands  a  "yes"  or  "no"  answer  (a  decision).  Begin  the 
question  with  words  such  as  "Can,"  "Does,"  or  "Is"  (e.g..  Can 
Nipper  equipped  units  achieve  and  maintain  a  level  of  training 
readiness  during  peacetime  and  provide  for  a  wartime  readiness 
capability  for  sustained  combat  operations?) 

DON'T  formulate  the  issue  as  an  investigative  in  nature 
question,  which  demands  an  analytical  answer,  by  beginning  the 
question  with  words  such  as  "How  well"  or  "What  is".  For 
e^tample,  contrast,  "How  well  does  the  Nipper  close  with,  detect, 
engage,  ..."  with  the  example  of  2.3.2.4a  above.  Note:  an 
investigative  issue  may  be  appropriate  for  an  AOIC,  since  they 
support  the  evaluation  and  not  the  decision. 
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c.  Few  issues.  DO  limit  to  a  few  by  focusing  on  the  total 
system  need  and  concerns  for  the  full  production  (MS  III) 
decision . 

DON'T  duplicate  coverage  by  overlapping  issues  (without  good 
reason) . 

DON'T  get  bogged  down  in  the  "eaches"  of  a  system  (e.g., 
elements  of  operational  effectiveness/suitability  and  ORD 
operational  characteristics) . 

d.  Apply  experiences.  DO  use  success  as  a  guide  (not  as  a 
rule) .  Apply  experiences  during  recent  COIC  approval  actions 
while  recognizing  system  differences. 

Section  III 
Defining  the  Scope 

3-9.  Characteristics 

The  scope,  by  definition,  is  a  statement  of  the  operational 
capabilities,  definitions,  and  conditions  which  focus  each  issue 
and  its  evaluation.  There  will  be  a  separate  scope  statement  for 
each  issue,  even  though  the  scope  for  the  second  or  successive 
issues  may  refer  to  and  expand  on  the  scope  statement  of  issue 
one.  The  scope  normally  begins  with  the  words,  "This  issue 
examines...,"  and  identifies: 

a.  Capabilities.  Operational  capabilities  to  be  examined 
(e.g.,  mission  accomplishment,  sustainment  training,  and/or 
combat  sustainment). 

b.  Definitions.  Special  terms,  either  system  peculiar 
reguire  definition  (e.g.,  system  description  ,  communication 
connectivity,  or  vehicle  payload)  or  measurement  peculiar  (e.g., 
start/stop  points  for  time  measures). 

c.  Conditions.  Evaluation  conditions,  including:  tactical 
context  and  scenario  (e.g.,  operational  mode  summary/mission 
profile.  Southwest  Asia  Standard  Scenario);  force  structure  and 
deployment  considerations  (e.g..  Doctrine  and  Organization  (D&O) 
Test  Support  Package  (TSP)  and  Corps/Division/Other  slice); 
Approved  threat  (e.g..  Threat  TSP  and  STAR);  crew  and  maintainers 
descriptions;  environmental  conditions  (natural  and  dirty 
battlefield) . 

d.  Other  data  sources.  When  an  issue  and  any  of  its  criteria 
reguire  technical  test  or  modeling/analysis  support. 

3-10.  Defining  the  Scope  -  Questions  to  Ask 

a.  What  are  the  operational  capabilities  of  concern? 
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b.  Do  force-on- force  operations  apply,  and,  if  so,  at  what 
level  (e.g.,  electronic  warfare  only  or  armored  force  in 
accordance  with  approved  threat  package  and  scenario)? 

c.  What  friendly  force  structure  and  operations  are 
necessitated  (e.g.,  single  system  only  or  force  slice;  crew  and 
maintainers;  approved  OMS/MP  and  scenario  or  only  elements 
thereof) ? 

d.  What  environments  apply  (e.g.,  natural  [terrain, 
visibility,  day/night,  climate]  and  battlefield  [MOPP  level, 
obscurants,  ECM,  etc.)? 

e.  What  terms  need  definition  (e.g.,  those  which  are  system, 
operation,  and  measurement  peculiar)? 

f.  Do  any  special  evaluation  methods  apply  (e.g.,  technical 
test  or  application  of  analytical  means)? 

3-11.  Defining  the  Scope  -  DO's  and  DON'Ts 

a.  Focus  issue.  DO  focus  evaluation  of  the  issue  by 
identifying  operational  capabilities  of  concern,  applicable 
operational  conditions,  applicable  definitions,  and  special 
evaluation  methodologies  (e.g.,  when  technical  test  or  analytical 
means  are  used  in  lieu  of  or  to  supplement  OT) . 

DON’T  specify  criteria  (i.e.,  characteristics  with 
performance  standards) . 

DON'T  specify  rationale  (i.e.,  justify  the  issue  or 
criteria) . 

DON'T  include  specific  conditions/definitions  better  suited 
as  part  of  a  criteria  (e.g.,  detection/engagement  envelope,  line 
of  sight,  pallet  weight  for  upload,  etc.). 

b.  Development  procedure.  DO  initially  prepare  the  scope  in 
draft  and  finalize  only  after  developing  applicable  criteria 
(i.e.,  selection  of  specific  criteria  may  in  fact  necessitate 
unique  conditions,  definitions,  or  evaluation  methodologies  not 
initially  anticipated). 

Section  IV 

Developing  the  Criteria 
3-12.  Characteristics 

Criteria  are,  by  definition,  those  measures  of  performance  which, 
when  achieved,  signify  that  the  issue  has  been  satisfied. 

Criteria  will  be  few  in  number,  but  there  will  be  at  least  one 
criteria  for  each  critical  issue.  Criteria  will: 
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a.  Be  focused.  COIC  focus  on  the  total  operational  system 
and  the  full  production  (MS  III)  decision,  even  though  they  may 
be  soft  for  MS  I  (e.g.,  "Will  be  capable  of  killing  tank  X" 
versus  "Will  have  a  50  percent  chance  of  killing  tank  X").  When 
firm  criteria  are  known  early,  they  will  be  stated  (e.g.,  "Will 
be  capable  of  roll-on  roll-off  transportation  by  C-130 
aircraft"). 

b.  Reflect  system  maturity.  COIC  are  formulated  without 
losing  sight  of  the  fact  that  the  "system"  is  in  a  constant  state 
of  development  (e.g.,  even  a  nondevelopmental  item  frequently 
does  not  have  mature  tactics,  techniques,  procedures,  and 
training  at  the  full  production  (MS  III)  decision). 

c.  Be  show  stoppers.  COIC  are  formulated  to  reflect  show 
stopper  measures  (e.g..  Believe  if  all  criteria  are  met,  the 
system  is  operationally  good  enough,  or,  to  the  contrary,  if  a 
criteria  is  not  met,  the  full  production  decision  will  not  be 
given).  Mandatory  Note  #2  (see  Section  VI,  this  Part)  is 
provided  to  avoid  use  of  criteria  as  automatic  pass/fail  measures 
during  evaluation  and  decision  making  such  that  other  credible 
evidence  of  an  operationally  effective  and  suitable  system  will 
be  considered  when  available  to  arrive  at  the  proper  decision. 

d.  Auditable  to  the  requirement  and  COEA.  This  does  not  mean 
that  criteria  are  a  direct  lift  from  these  documents,  but  that 
they  are  traceable  to  specific  requirements  and  findings  of  these 
documents  by  rationale. 

NOTE:  Criteria  are  categorized  as  "derived"  when  they  are 

developed  by  combining  two  or  more  requirements  into  a  single 
higher  order  of  measure  or  drawn  from  sources  (e.g.,  the  COEA) 
other  than  the  requirement  to  provide  specific  measures  of 
performance  not  provided  in  the  requirement  document  (e.g.,  the 
ORD  requires  improved  survivability  whereas  cost  and  COEA  data 
support  a  need  for  20  percent  more  combat  capable  systems). 

3-13.  Criterion  statement 

Figure  3-7  depicts  the  major  elements  of  a  criterion  statement, 
each  of  which  must  be  addressed,  and  presents  an  example  of  a 
properly  constructed  criterion  statement,  with  explanations  for 
the  specific  wording.  Special  emphasis,  when  applicable,  must  be 
devoted  to  choosing  which  type  of  total  system  (individual  or 
unit)  is  to  be  examined  and  whether  the  characteristic  of 
interest  is  a  performance  standard  or  a  baseline  comparison. 
Additionally,  the  following  must  be  considered:  criteria  mature 
with  the  operational  requirement  (soft  for  MSI  and  firm  for  MS 
II);  system  (hardware,  software,  tactics,  techniques,  and 
procedures,  etc.)  still  maturing  at  MS  III;  information  available 
from  the  requirement  document  (lack  of  specificity  in  performance 
parameters  may  increase  the  potential  for  evaluation  bias  and 
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thereby  dictate  use  of  baseline  comparison);  the  acquisition 
objective  (cost  may  override  performance  and  the  criteria 
therefore  reflect  current  system  performance).  As  reflected  in 
the  figure,  there  are  choices  for  each  element,  wherein  the 
correct  choice  is  system/situation  dependent  (e.g.,  a  tank  and  a 
communications  system  will  have  differently  structured  criteria) . 

Figures  15  and  16  present  additional  system/situation  examples 
of  characteristics  of  interest  and  typical  means  of  measurement 
(They  are  not  complete  criteria  statements). 

*********>***insERT  figure  3-7****************** 

As  an  illustration  of  proper  structure,  consider  the  criterion 
statement,  "The  tank  will  kill  50%  more  enemy  armored  vehicles  at 
ranges  out  to  3  kilometers." 

The  object  to  be  examined  is  the  tank. 

The  characteristic  of  interest  is  "kill  armored  vehicles," 
which  constitutes  a  critical  performance  capability,  and  the 
qualifier  "more"  alludes  to  a  comparison  with  a  baseline. 

The  magnitude  of  50%  is  quantitative  and  the  direction  "at 
least"  is  implied. 

The  constraint  condition  of  "out  to  3  kilometers"  is  both 
operational  and  tight,  and  "enemy"  implies  battlefield 
conditions. 

The  scoring  criteria  is  kills,  which  would  be  based  on 
definitions  (e.g.,  mobility,  firepower,  catastrophic) 

NOTE:  A  caution  on  constraint  conditions  -  they  must  be 
operationally  realistic.  If,  for  example,  their  interpretation 
allows  for  use  of  unrepresentative  threat  or  friendly  operations 
in  test  and  evaluation,  they  have  been  improperly  stated. 

**************INSERT  figure  3-8******************* 

**************INSERT  figure  3-9******************* 

a.  Total  system.  As  indicated  earlier,  special  emphasis  must 
be  placed  on  choosing  the  correct  total  system  -  an  individual 
system  or  an  organizational  unit  -  to  be  the  object  examined  (see 
Figure  3-10).  Factors  which  would  lead  to  selection  of  a  single 
system  include:  technical  criteria  (e.g.,  ascend/descend  a  60 
degree  slope);  the  system  operates  and/or  is  employed  as  an 
independent  system  (tractor  and  trailer);  or  the  purpose  of  the 
acquisition  is  to  benefit  the  system  alone  (e.g.,  larger  caliber 
tank  main  gun).  Factors  which  would  lead  to  selection  of  an 
organizational  unit  include:  the  purpose  of  the  acquisition  is  to 
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benefit  a  unit  (e.g.,  an  automatic  detection  and  defense  system 
authorized  one  to  a  platoon  to  improve  survivability);  the  system 
operates  and/or  is  employed  as  an  element  of  a  unit  (e.g.,  an  air 
defense  system  -  fire  unit  -  which  receives  target  cuing, 
hand-off,  and  engagement  responsibilities/authority  from  the  next 
higher  element  -  platoon) ;  the  system  represents  a  system  of 
systems  (e.g.,  a  force  level  communications  system  made  up  of 
multiple,  dissimilar  subsystems  -  ATCCS);  or  a  concern 
(characteristic  of  interest)  which  requires  a  unit  measure  (e.g., 
more  combat  capable  vehicles  remaining) . 

*************INSERT  figure  3-10**************** 

b.  Performance  standard  versus  baseline  comparison  criteria. 
Also  as  indicated  above,  special  emphasis  must  be  placed  on 
determining  whether  the  characteristic  of  interest  can  be  stated 
as  a  performance  standard  or  will  require  baseline  comparison. 
Most  characteristics  of  interest  will  be  stated  as  performance 
standards.  However,  two  key  situations  will  dictate  use  of 
baseline  comparison:  the  system  is  a  replacement  system  or  a 
materiel  change  to  an  existing  system  and  requirements  documents 
or  other  sources  fail  to  provide  an  adequate  basis  for  deriving 
performance  standards;  the  independent  operational  evaluator 
identifies  and  justifies,  to  the  satisfaction  of  the  CBTDEV 
proponent/FP,  sufficient  risk  of  bias  in  T&E.  Although  this  is  a 
break  with  the  past,  when  baseline  comparison  was  reserved  for 
exceptional  cases  and  then  only  when  absolutely  necessary, 
baseline  comparison  is  now  encouraged  in  the  situations  outlined. 
It  should  be  kept  in  mind,  however,  that  the  use  of  baseline 
comparison  criteria  results  in  side  by  side  comparison  testing  to 
support  evaluation  of  the  system.  The  criticality  of  this 
approach  to  the  evaluation  effort  must  therefore  be  sufficiently 
high  to  justify  the  expenditure  of  significant  additional 
resources . 

3-14.  Developing  the  criteria  -  DOs  and  DON'Ts 

NOTE:  Each  DO  is  followed,  when  appropriate,  by  one  or  more 
companion  DON'Ts. 

a.  Minimum  need.  DO  focus  on  the  minimum  needed  for  full 
production  (MS  III)  decision  -  discard  or  revise  if  a  shortfall 
would  not  be  a  "show  stopper." 

DON'T  include  "desired"  characteristics. 

DON'T  specify  "firm"  criteria  for  MS  I  TEMP  unless  known 
to  be  stable  (e.g.,  transportable  by  CH-47D) . 
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DON'T  embed  peripheral  issues  in  criteria  to  ensure 
evaluation  (e.g.,  the  training  program  must  be  the  optimum 
training  strategy) . 

b.  Measures  of  performance.  DO  use  measures  of  performance 
which  undergird  the  system's  operational  effectiveness  and 
suitability  in  terms  of  critical  combat  missions  to  be 
accomplished. 

DON'T  use  measures  of  effectiveness  such  as  Force  Exchange 
Ratio  (FER),  Loss  Exchange  Ratio  (LER),  or  other  COEA  measures 
which  depend  on  large  scale  modeling  beyond  the  capability  of  the 
operational  evaluation  (operational  test  does  not  provide  enough 
trials  or  steady  state  operations  to  revisit  or  substantiate  the 
COEA) . 


c.  Qualitative  criteria.  DO  specify  qualitative  criteria 
(which  must  be  measurable)  only  when  quantitative  criteria  are 
not  applicable. 

DON'T  specify  a  confidence  level. 

d.  Test  and  evaluation  limitation.  DO  specify  measures 
without  (unconstrained  by)  consideration  of  the  applicable 
test /evaluation  methodology  to  be  used  for  resolution. 

DON'T  exclude  a  critical  criteria  because  it  can  only  be 
answered  by  technical  test  or  modeling  (criteria  focus  the 
operational  evaluation  and  the  decision,  not  a  test). 

DON'T  compromise  criteria  to  accommodate  test  and  evaluation 
frailties  (i.e.,  T&E  instrumentation,  facilities,  or  other 
resources  should  not  restrict  the  criteria  if  it  is  deemed 
critical ) . 

e.  Probabilistic  measures.  DO  specify  soldier-machine 
measures  in  probabilistic  terms,  however  they  must  be  realistic 
(e.g.,  use  the  median  if  a  high  degree  of  performance  is  not 
needed,  or  80/90%  if  a  high  degree  is  needed 

DON'T  specify  or  imply  100%  performance  when  the  operation 
must  be  accomplished  by  soldiers. 

f.  Conditions  and  definitions.  DO  specify  the  conditions  and 
definitions  needed  for  evaluation  (e.g.,  the  operational 
constraint  (engagement  envelope)  and/or  scoring  criteria 
(stop/start  point  for  a  time  line,  destroy/kill  definition, 

etc . ) ) . 

DON'T  leave  ambiguities  which  can  result  in  erroneous  T&E  of 
the  criteria  (e.g.,  don't  say  "more  survivable"  because 
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survivability  can  be  measured  as  either  more  combat  vehicles 
remaining  at  a  given  point  in  time  or  as  more  threat  kills 
because  the  vehicle  remains  combat  capable  longer) . 

DON'T  over  specify  constraints  and  definitions  (e.g.,  a 
constraint  allowing  operation  only  in  temperatures  above  70 
degrees  Fahrenheit  would  not  support  world  wide  deployment;  the 
engagement  constraint  "targets  entering  the  crew's  fire  zone" 
could  be  operationally  limited  by  terrain  rather  than  the  range 
capability  of  a  direct  fire  weapori) . 

g.  Total  system  measures.  DO  specify  total  system  measures 
(e.g.,  operator  load  vehicle,  accomplish  OMS/MP  at  stated  speeds, 
C-130  roll-on/off,  etc.) 

DON'T  specify  component  measures  (e.g.,  materiel/software 
performance,  human  factors  constraints,  technical  standards, 
etc. ) 


h.  Lowest  level  system.  DO  specify  the  lowest  level  system 
possible  and  appropriate  (the  preference  is  a  single  system  but, 
when  required,  an  organizational  level  may  be  more  appropriate) 
(e.g.,  the  Paladin  (M109A6)  used  the  individual  howitzer  for 
mission  accomplishment  and  the  battalion  for  survivability; 
communications  systems  normally  use  nets  for  mission 
accomplishment  and  key  components  for  set-up/tear-down  time; 
trucks  are  typically  addressed  without  trailers,  etc.). 

DON'T  measure  a  structure  which  obscures  performance  of  the 
system  of  concern  (e.g.,  an  improvement  to  a  portion  of  a  fleet 
of  vehicles  may  bring  major  improvement  to  the  vehicle,  moderate 
improvement  to  the  platoon,  and  only  slight  improvement  to  the 
combined  arms  team) . 

i.  Higher  order  measures.  DO  specify  higher  order  measures 
(e.g.,  P^/percent  target  kill,  percent  messages  sent  and 
received,  A^^/maintenance  ratio/MTBOMF  (but  not  all  three),  etc.) 

DON'T  specify  eaches  (e.g.,  probability  of  detection, 
identification,  hand-off,  engagement,  hit,  kill  given  a  hit, 
connectivity,  messages  received,  etc.;  RAM  characteristics). 

j.  Baseline  comparison.  DO  specify  baseline  comparison 
criteria  only  when  appropriate  (see  2.3.4.2b)  and  state  an 
improvement  percentage  when  the  acquisition  objective  is  improved 
performance  and  the  end  result  will  be  higher  system  cost. 

DON'T  state  an  improvement  percentage  for  baseline 
comparison  when  cost  benefit  is  the  reason  for  the  acquisition. 
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DON'T  use  statistical  significance  as  rationale  for  the 
stated  improvement  percentage. 

k.  Quantitative  criteria.  DO  use  quantitative  criteria, 
which  are  preferred,  when  possible. 

DON'T  use  qualitative  criteria  unless  quantitative  criteria 
cannot  be  developed  or  are  not  applicable. 

l.  Lessons  learned  (recent  experiences).  DO  apply  lessons 
learned  from  previous  evaluations  to  avoid  pitfalls. 

DON'T  allow  duplicate  or  overlapping  criteria  (i.e.,  a 
system  should  not  be  placed  in  double  jeopardy  for  a  single 
shortcoming) . 

Section  V 

Providing  the  Rationale 
3-15.  Characteristics 

The  rationale,  by  definition,  provides  justification  for  the 
criteria,  not  the  issue,  and  an  audit  trail  to  the  requirement 
specified  in  the  MNS,  ROC/ORD,  FD,  COEA,  system  specification, 
etc..  It  states  the  reason  for  selecting  a  particular 
characteristic  or  capability  and  identifies,  by  document  and 
paragraph,  the  source  of  the  information.  In  the  case  of  derived 
criteria,  the  rationale  will  provide  the  basis  and  methodology 
used . 

3-16.  Providing  the  rationale  -  Questions  to  Ask 

a.  References.  Are  appropriate  source  references  included 
for  all  criteria?  Is  there  an  ORD  paragraph  reference  for  each 
criterion  stated? 

b.  Derived  criteria.  Are  the  basis  and  methodology  discussed 
for  all  "derived"  criteria  (e.g.,  probability  of  kill 
incorporates  probabilities  of  detection,  identification, 
engagement,  hit,  and  kill  given  a  hit)? 

c.  COEA  relationship.  Is  the  relationship  between  the 
criteria  and  COEA  results  addressed  where  applicable  (e.g.,  the 
ORD  requires  improved  survivability  (over  that  of  the  baseline 
system)  and  the  COEA  identifies  a  minimum  requirement  for  20% 
more  combat  capable  systems)? 

3-17.  Providing  the  rationale  -  DOs  and  DON'Ts 

a.  Criteria  justified.  DO  provide  a  complete  justification 
for  each  criteria. 
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DON'T  justify  the  issue. 

DON'T  inject  new/additional  criteria  into  the  rationale. 

b.  Criteria  audit  trail.  DO  establish  a  complete  audit  trail 
by  indicating  the  specific  document  and  paragraph  within  the 
document  from  which  the  requirement  was  drawn.  Every  criteria 
must  have  basis  in  the  requirement  document  (ORD/ROC).  This  does 
not  mean  that  it  must  be  a  direct  lift. 

c.  Critical  mission  justification.  DO  justify  why  a 
particular  mission  or  use  was  selected  when  multiple  missions  or 
uses  are  possible. 

Section  VI 

Establishing  the  Notes 
3-18.  General 

Mandatory  notes  and  any  other  required  notes,  explanations,  or 
definitions  will  be  included  after  the  last  issue  set.  They 
serve  to;  emphasize  the  purpose  and  scope  of  COIC,  in  relation  to 
the  full  set  of  OIC;  place  T&E  results  related  to  COIC  in  the 
proper  perspective;  discuss  lengthy  T&E  conditions  or 
definitions . 

3-19.  Mandatory  note  #1 

a.  The  note.  Provided  the  following  note  modified  to  reflect 
appropriate  characteristics  applicable  for  the  specific  system 
(e.g.  if  a  maintenance  ratio  included  as  a  criteria  then  RAM  may 
not  apply  to  this  note) : 

"Note  #  1.  Criteria  X.  Y.  and  Z  are  total  system  measures.  As 
such  they  inherently  cover  hardware,  software,  personnel, 
doctrine,  organization,  and  training.  System  individual 
characteristics  of  operational  capability,  survivability,  RAM, 
organization,  doctrine,  tactics,  logistics  support,  training,  and 
MANPRINT  (which  includes  the  domains  of  manpower,  personnel, 
training,  human  factors  engineering,  system  safety,  and  health 
hazards)  related  to  these  criteria  will  be  provided  bv  the 
operational  independent  evaluator  in  the  Operational  Test  and 
Evaluation  Plan  fTEP)." 

a.  Discussion  of  note  #1. 

(1)  This  note  serves  to  emphasize  to  the  COIC  developer 
that  total  system  measures  are  preferred. 

(2)  This  note  acknowledges  that  some  criteria  will  not  be 
total  system  measures,  and  identifies  for  the  evaluator  and 
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reviewers  those  designated  criteria  (X,  Y,  and  Z)  which  are  in 
fact  total  system  measures. 

(3)  This  note  commits  to  addressing  the  more  detailed 
system  individual  characteristics  in  the  operational  TEP. 

3-20.  Mandatory  note  #2 

a.  The  note.  Provide  the  following  note: 

"Note  #  2.  Criteria  are  not  provided  as  automatic  (default) 
pass/fail  measures.  Rather  they  represent  estimates  of 
performance  for  which  a  breach  would  require  a  careful  senio_r 

level  management  reassessment  of  cost  effectiveness _ and  program 

options  during  the  program  milestone  decision  review.*’ 

b.  Discussion  of  note  #  2. 

(1)  This  note  emphasizes  that  criteria  are  not  "automatic" 
pass/fail  measures. 

(2)  This  note  highlights  the  fact  that  breach  of  a  criteria 
constitutes  a  show  stopper  until  convincing  evidence  can  be 
presented  to  decision  makers  that  the  program  should  proceed  in 
spite  of  the  shortfall.  "Convincing  evidence"  might  include  a 
revised  risk  assessment,  specific  observations  and  data  from 
operational  tests,  baseline  comparison  data,  COEA  updates,  or  a 
revised  threat  assessment. 

3-21.  Mandatory  note  #3. 

a.  The  note.  Provide  the  following  note  for  those  COIC 
applicable  to  the  MS  I  TEMP: 

"Note  #  3.  These  COIC  are  derived  from  the  user’s  initial 
requirements  for  the  system.  These  COIC  will  be  updated  prior  to 
MS  II  based  on  the  revised  ORD  and  final  updated  COEA." 

b.  Discussion  of  note  #3, 

(1)  This  note  is  applicable  only  for  COIC  in  support  of  the 
TEMP  approved  in  advance  of  Milestone  I . 

(2)  This  note  highlights  the  fact  that  COIC  for  the  MS  I 
TEMP  may  contain  "soft"  criteria  which  will  be  updated  as  the 
system  matures. 

3-22.  System  peculiar  notes 

System  peculiar  notes  are  those  necessary  for  understanding 
They  will  commonly  focus  on  definitions  or  lengthy  test  and 
evaluation  conditions. 
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Section  VII 

COIC  Checklist  and  Development  Sample 
3-23.  Checklist  for  COIC 

Figure  3-11  is  a  COIC  checklist  for  use  by  COIC  preparers  and 
staffers  at  all  levels.  The  checklist  covers  both  content  and 
processing  events.  Materiel  and  IMA  systems  are  covered. 

3-24.  COIC  development  sample 

Figure  3-12  is  a  COIC  development  sample.  There  are  two  parts  - 
the  situation  and  the  solution.  The  situation  provides 
applicable  operational  requirements  Information,  program  status, 
similar  system  recent  experience,  and  acquisition  strategy.  The 
solution  provides  a  resultant  set  of  COIC  for  the  situation 
described  applying  the  guidelines  presented  in  this  guide.  There 
are  other  possible  solutions  but  note  that  this  has  been 
successful. 

************INSERT  FIGURE  3-11*************** 

************INSeRT  figure  3-12*************** 
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EVENT 


CBTDEV  APPROVED  ORD 
DA  APPROVED  ORD 
CBTDEV  APPROVED  COIC 
DA  APPROVED  COIC 
COIC  TO  MATCEV 
MATDEV  APPROVED  TEMP 
DA  APPROVED  TEMP 


ACAT  I /II  MAT, 
T/T  MAISRC  AIS,  & 
OSD  T&E  OVERSIGHT 
NLT  DAYS  PRIOR  TO 


MS  I 

MS  II 

185 

230 

125 

170 

95 

140 

N/A 

100 

95 

95 

65 

65 

45 

45 

ACAT  III /IV  MAT  & 
NON-MAISRC  T/T  AIS 
NLT  DAYS  PRIOR  TO 


MS  I 

MS  II 

120 

120 

N/A 

N/A 

75 

75 

N/A 

N/A 

75 

75 

45 

45 

N/A 

N/A 

Table  2-1.  Synchronized  Process  Schedules:  MS  I/II 


NLT  DAYS  PRIOR  TO 

EVENT 

DA  TEMP  APPROVAL 

CBTDEV  APPROVED  RQMT  CHANGE 

185 

DA  APPROVED  RQMT  CHANGE 

125 

CBTDEV  APPROVED  COIC 

95 

DA  APPROVED  COIC 

55 

COIC  TO  MATDEV 

50 

MATDEV  APPROVED  TEMP 

20 

DA  APPROVED  TEMP 

0 

Table  2-2.  Synchronized  Process  Schedule:  Post-MS  II  Changes 
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EVENT 


NLT  DAYS  PRIOR  TO 
MDR  I 


FP  APPROVED  COIC 
SAIS-AE  APPROVED  COIC 
VDISC4  APPROVED  COIC 
COIC  TO  PM/TIWG 
MATDEV/SFTDEV  APPROVED  TEMP 
DUSAfOR)  APPROVED  TEMP _ 


Table  2-3.  Synchronized  Process  Schedule; 
Strategic  and  Sustaining  Base  (S/SB)  IMA  COIC/TEMP 
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ATCD-ZZ  (71-3) 


U.S.  Army  Training  and  Doctrine  Command 

^  o  *  17 


S:  ZZZZZZZZ 


MEMORANDUM  FOR  CDR  OPTEC,  ATTN;  CSTE-ZA  (EVALUATOR'S  NAME) 

PM 

HQDA,  ATTN;  DAMO-FDR 

TRADOC  MATERIEL  EVALUATION  COMMITTEE  (TMEC) 

SUBJECT;  Critical  Operational  Issues  and  Criteria  (COIC)  for  X 
System 

1.  References; 

a.  Memorandum,  HQDA,  DUSA(OR)  and  ADCSOPS-FD,  3  Jun  92, 
subject;  Critical  Operational  Issues  and  Criteria  (COIC). 

b.  Message,  HQDA,  DAMO-FDZ,  091515Z  May  91,  subject;  HQDA 
Approval  Procedures  for  Materiel  System  Critical  Operational 
Issues  and  Criteria  (COIC). 

2.  This  memorandum  forwards  final  draft  COIC  for  subject  system 
for  your  concurrence.  Per  references  la  and  b  ^^ove,  this 
constitutes  TRADOC  position  staffing  to  recommend  that  DCSCD 
approve  and  forward  these  COIC  to  DA  for  ADCSOPS-FD  approval . 
Request  your  position  be  provided  this  headquarters  (ATTN; 
ATCD-M/G/S)  not  later  than  (two  weeks) —  .  This  will  support 
TEMP  approval  by  f date )  as  currently  scheduled. 

3.  Significant  changes  will  be  staffed  with  you  before  DCSCD 
approves  the  COIC  for  TRADOC.  An  expedited  staffing  technique 
will  be  used  to  maintain  current  approval  schedule. 

4.  TRADOC  POC  is  YYYYYYYYYYYY . 

FOR  THE  COMMANDER; 


1  End 


XXXXXXXXXXXXXXXXXXXXXXXXX 
Major  General,  GS 
Deputy  Chief  of  Staff  for 
Combat  Developments 


TRADOC  COMMAND/CENTER/ SCHOOL,  ATTN;  PCD  AND  TSM _ 

Figure  2-8.  Sample  Combat  Developer  Final  COIC  Staffing 
Memorandum  —  Before  Forwarding  to  DA  for  Approval 


j-ys 


U.S.  Army  Training  and  Doctrine  Command 
ATCD-ZZZ  (71-3) 

MEMORANDUM  THREW  COMMANDER,  U.S.  ARMY  OPERATIONAL  TEST  AND 
EVALUATION  COMMAND,  ATTN;  CSTE-ZA,  PARK  CENTER  IV  4501  FORD 
AVENUE,  ALEXANDRIA,  VA  22302-1458 

FOR  HQDA,  ATTN:  DAMO-FDR,  WASHINGTON,  D.C.  20310-0400 

SUBJECT:  Critical  Operational  Issues  and  Criteria  (COIC)  for  "X" 
System 

1.  References; 

a.  Memorandum,  HQDA,  DUSA(OR)  and  ADCSOPS-FD,  3  Jun  92, 
subject:  Critical  Operational  Issues  and  Criteria  (COIC). 

b.  Message,  HQDA,  DAMO-FDZ,  091515Z  May  91,  subject:  HQDA 
Approval  Procedures  for  Materiel  System  Critical  Operational 
Issues  and  Criteria  (COIC). 

2.  This  memorandum  forwards  TRADOC  approved  COIC  for  subject 
system  (Enel  1)  for  ADCSOPS-FD  approval  per  references  la  and  b. 
These  COIC  were  previously  staffed  with  and  concurred  in  by  PM 

and  OPTEC .  (If  there  is  an  unresolved  difference  of 

position,  it  should  be  described  here.)  .... 

3.  TRADOC  POC  is  YYYYY YYYYYYY . 

FOR  THE  COMMANDER: 


1  Enel  XXXXXXXXXXXXXXXXXXXXXXXXX 

Major  General,  GS 
Deputy  Chief  of  Staff  for 
Combat  Developments 

CF: 

HQDA,  ATTN:  DAMO-FDR  (ADVANCE  COPY) 

OPERATIONAL  EVALUATION  COMMAND,  ATTN:  Evaluator  Office 
PM 

TRADOC  CENTER/ SCHOOL,  ATTN;  DCD  AND  TSM 


Figure  2-9.  Sample  Combat  Developer  Memorandum  Forwarding  COIC 

to  DA  for  Approval 
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Figure  2-17.  COIC-ORD  Crosswalk  Horse  Blanket  Pages 


HARDWARE 


H  H 
<  CO 
LU  OC  >- 

X  LLl  CO 
I—  ^ 

O 


ojlo 


co[S 

co2q 

OZLU 

p  ^ 

qOO 
<  LU  QC 
PHD- 


TRAINED  CREW 


Critical  Operational  Issues  and  Criteria 

for 

the  "X"  System  (or  "Y"  Modification  to  the  "X"  System) 
for  Test  and  Evaluation  Master  Plan  Supporting 
Milestone  "Z"  (I/II/III)  or  Modification  Approval  Package 


1.0  Issue;  (See  Section  II) 

1.1  Scope;  (See  Section  III) 

1.2  Criteria;  (See  Section  IV) 

1.2.1 

A  dendritic  numbering  system  is 

1.2.2  used  to  standardize  format. 

1.2. n 

1 . 3  Rationale; 

1.3.1 

1.3.2 

1. 3. n 


Subsequent  issue  sets  are  numbered 
2  through  n.  NOTE:  the  total  is 
commonly  six  (6)  or  less,  with 
three  (3)  being  the  norm. 


Note  1;  (mandatory)  (See  Section  VI) 

Note  2;  (mandatory)  (See  Section  VI) 

Note  3:  (mandatory  for  MS  I  TEMP  COIC)  (See  Section  VI) 
Notes  4  through  n  (system  peculiar  -  see  Section  VI) 


Figure  3-2.  COIC  Format 


(See  Section  V) 


Rationale  subparagraphs  correspond 
to  those  of  each  criteria. 
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Figure  3-3.  COIC  Relationships 
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Figure  3-6.  COiC  Relationship 
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Figure  3-7.  A  Criterion  Statement 


SITUATION  MEASURE 
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SITUATION  MEASURE 


Figure  3-9.  Characteristics  of  Interest  -  Sustainment  Examples 


Checklist  for 

Critical  Operational  Issues  and  Criteria  (COIC) 


NOTE:  This  checklist  is  provided  to  be  followed  as  a  guideline 
for  use  by  all  involved  in  the  preparation,  review,  and  approval 
of  Critical  Operational  Issues  and  Criteria  (COIC) • 
questions  are  intended  to  be  answered  "yes."  If  a  question  is 
answered  "no,"  the  applicable  element  should  be  reworked  or 
justification  provided.  Approved  COIC  are  included  in  the  Test 
and  Evaluation  Master  Plan  (TEMP). 

1.  COIC  Format  and  Content. 

a.  Heading. 

(1)  Does  it  state  "Critical  Operational  Issues  and  Criteria 

for"? 

(2)  Does  it  contain  the  system  name? 

(3)  Does  it  identify  the  applicable  TEMP? 

b .  Format . 

(1)  Is  there  a  Scope,  Criteria,  and  Rationale  paragraph  for 
each  issue? 

(2)  Does  paragraph  numbering  follow  the  dendritic  format  of 
X.O  -  Issue,  X.l  -  Scope,  X.2  -  Criteria,  and  X.3  -  Rationale? 

(X  is  the  issue  number,  i.e.,  1,  2,  etc.) 

(3)  Does  each  criteria  have  an  associated  rationale 
subparagraph? 

(4)  Are  the  mandatory  notes  and  other  system  peculiar  notes 
included? 

c.  Content  -  Issues. 

(1)  Do  the  issues  reflect  only  those  few  key  operational 
concerns  and  standards  for  determining  the  system's  readiness  at 
the  "Beyond  LRIP"  (MS  III)  decision  review  to  enter  full  scale 
production? 


Figure  3-11.  Checklist  for  COIC 


(2)  Are  the  Issues  in  the  form  of  questions  to  be  answered 
"yes"  or  "no"  (i.e.,  no  issue  should  be  investigative  in  nature  - 
"How  well"  or  "What  is")? 

(3)  Are  the  issues  based  on  the  MNS? 

(4)  Are  the  issues  operationally  realistic  and  do  they  ask 
if/whether  a  task/function  or  mission  (kill,  be  sustained,  etc.) 
can  be  achieved? 

(5)  Do  the  issues  focus  on  the  total  operational  system  and 
not  its  component  parts? 

(6)  Do  the  issues  focus  the  decision?  (They  should  not 
over  generalize,  e.g.,  "Is  system  X  operationally 
effective/sustainable  in  an  operational  environment.) 

(7)  Are  issue  statements  free  of  criteria  (i.e., 
performance  standards ) ? 

(8)  Has  overlapping  coverage  between  issues  been  avoided  to 
the  degree  possible  and  appropriate? 


d.  Content  -  Scope. 

(1)  Does  the  scope  identify  the  operational  capabilities  to 
be  examined? 

(2)  Are  terms  peculiar  to  the  system  and  evaluation  of  each 
issue  defined? 

(3)  Are  the  tactical  context  and  scenario(s)  applicable  to 
evaluation  of  each  issue  identified? 

(4)  Are  key  system  deployment  and  organizational  structure 
factors  applicable  to  evaluation  identified? 

(5)  Are  applicable  approved  threat  documents  referenced? 

(6)  Are  applicable  crew  and  maintainers  identified? 

(7)  Are  key  natural  and  battlefield  environments 
identified? 

(8)  Have  requirements  for  technical  testing  and  modeling 
analysis  been  identified? 
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(9)  Is  the  scope  free  of  criteria  and  requirements 
statements  (e.g.,  must  do  something)? 

(10)  Is  the  scope  free  of  requirements  for  statistical 
confidence  levels  applicable  to  the  criteria? 

e.  Content  -  Criteria. 

(1)  Is  there  at  least  one  criteria  for  each  critical 
operational  issue? 

(2)  Is  each  criteria  a  "show  stopper"  for  the  "Beyond  LRIP" 
(MS  III)  decision? 

(3)  Do  the  criteria  represent  a  performance  threshold 
(e.g.,  quicker  delivery  of  mission/operational  orders  [MS  I  TEMP] 
or  delivery  of  mission/operational  orders  within  one  hour,  on  the 
average,  after  initiation  of  operations  [MS  II  TEMP]? 

(4)  Are  all  criteria  based  on  or  derived  from  requirements 
documented  in  the  MNS,  ROC/ORD,  FD,  COEA,  etc.,  and  do  they 
reflect  the  critical  operational  needs  and  constraints?  (The 
criteria  do  not  have  to  be  a  direct  lift  but  must  be  auditable  to 
an  approved  source  document.) 

(5)  Do  the  criteria  reflect  a  level  of  system  maturity 
appropriate  to  the  milestone  TEMP? 

(6)  Has  overlapping  coverage  among  criteria  been  avoided  to 
preclude  multiple  failure  for  a  single  shortfall? 

(7)  Are  all  criteria  which  are  not  total  system  measures 
(the  preference)  fully  justifiable? 

(8)  Do  criteria  reflect  only  essential  operational 
requirements  (not  desired  capabilities)? 

(9)  Wherever  possible,  are  higher  order  measures  of 
performance  (e.g.,  probability  of  kill,  probability  of  successful 
communications,  etc.)  stated  rather  than  those  of  contributing 
components  (e.g.,  probabilities  for  detecting,  engaging,  hitting, 
and  killing  a  target;  probabilities  for  connectivity  message 
accuracy,  RAM,  etc.)? 

(10)  Do  the  criteria  avoid  the  use  of  Force  Exchange  Ratio 
(FER),  Loss  Exchange  Ratio  (LER),  or  similar  operational 
effectiveness  measures  more  appropriate  for  COEA/modeling? 
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(11)  Is  a  baseline  comparison  used  only  when  a  specific 
performance  measure  cannot  be  derived,  when  directed  by  higher 
authority,  or  to  reduce  the  chance  of  bias  during  test  and 
evaluation? 

(12)  If  a  baseline  comparison  is  used,  and  performance 
improvement  is  the  objective,  is  an  improvement  percentage 
specified? 

(13)  Are  qualitative  criteria  measurable? 

(14)  Are  all  constraint  conditions  applicable  to  evaluation 
of  each  criteria  stated  and  consistent  with  the  scope  (e.g., 
MOPP-IV,  electronic  warfare,  etc)?  (Note;  they  may  also  be 
included  in  the  system  peculiar  notes.) 

(15)  Are  all  definitions  applicable  to  evaluation  of  each 
criteria  stated  and  consistent  with  the  scope  (e.g.,  firepower 
kill,  payload,  etc.)?  (Note;  they  may  also  be  included  in  the 
system  peculiar  notes.) 

(16)  Have  potential  ambiguities  which  could  result  in 
erroneous  T&E  been  avoided? 

(17)  Are  probabilistic  criteria  used  when  man-machine 
interface  dependent  (e.g.,  X%  of  attempts,  median  time,  etc.)? 

(18)  Is  the  appropriate  level  system  (i.e.,  individual 
system,  a  team,  platoon,  etc.)  addressed  by  each  criteria? 
(Criteria  must  be  the  lowest  level  appropriate  for  the  system  - 
an  individual  system  is  preferred;  an  organizational  element 
should  be  used  when  the  system's  primary  mission  contributes  to 
unit  performance  or  the  measure  of  performance  demands  a  unit, 
e.g.,  more  operational  systems  remaining.) 

(20)  Are  all  measures  of  performance  critical  to  the 
"Beyond  LRIP"  (MS  III)  decision  covered?  (No  key  criteria  should 
be  excluded  because  the  data  source  was  other  than  operational 
test  or  problems  collecting  needed  data  were  anticipated.) 

(21)  Are  criteria  free  of  confidence  levels? 
f.  Content  -  Rationale. 

(1)  Do  the  rationale  statements  justify  each  criteria? 
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(2)  Are  reasons  statedf or  selecting  the 
characteristic/capability  used? 

(3)  Are  the  ORD,  MNS  (for  AIS)/  and/or  other  source 
document  paragraph  references  identified? 

(4)  Are  complete  references  provided  for  criteria  derived 
by  combining  characteristics  or  capabilities? 

(5)  Is  an  audit  trail  to  the  COEA  provided? 
g.  Content  -  Notes. 

(1)  Are  mandatory  notes  #1  and  #2  present? 

(2)  Have  total  system  criteria  been  identified  in  mandatory 
note  #1? 

(3)  Is  mandatory  note  #3  present  for  COIC  in  support  of  the 
MS  I  TEMP? 

(4)  Are  notes  peculiar  to  the  system,  as  referenced  in  the 
body  of  the  COIC,  provided? 

2.  COIC  Review  and  Approval  -  ACAT  I/II  Materiel, 
Theater/Tactical  MAISRC,  OSD  T&E  Oversight,  and  Other  Systems 
Requiring  Approval  by  HQDA,  DCSOPS-FD. 

a.  For  TRADOC  DCSCD  approval; 

(1)  Is  the  ORD  DA  approved? 

(2)  Is  the  following  coordination  complete; 

Proponent  “  coordination  with  the  MATDEV  and  OPTEC 

OEC? 

(b)  HQ  TRADOC  -  coordination  with  the  TMEC  (TRADOC  Reg 
15-3),  MATDEV,  CG  OPTEC,  and  DAMO-FDT  A/0? 

(3)  Have  all  concurred  with  the  COIC?  (If  "NO,"  strong 
rationale  must  be  provided  for  TRADOC  DCSCD  consideration.) 

b.  For  DA  ADCSOPS-FD  approval; 

(1)  Are  the  COIC  TRADOC  DCSCD  approved? _ 
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(2)  Has  the  CG,  OPTEC  concurred  with  the  COIC? 


(3)  If  the  CG,  OPTEC  nonconcurred  and  TRADOC  DCSCD 
disagrees  with  the  nonconcurrence,  has  a  joint  CG  OPTEC/TRADOC 
DCSCD/DA  ADCSOPS~FD  forum  been  set  for  resolution? 

(4)  Has  the  COIC-ORD  cross-walk  "Horse  Blanket"  been 
prepared  for  the  DA  decision  briefing? 

(5)  Have  the  appropriate  DA  Staff  elements  concurred  with 
the  COIC? 


(6)  Has  the  appropriate  ODCSOPS  pre-brief  been 
accomplished? 

3.  COIC  Review  and  Approval  -  ACAT  III/IV  Materiel  and 
Theater/Tactical  non-MAISRC  Systems  not  on  the  OSD  T&E  Oversight 
List. 


a.  Is  the  ORD  -  TRADOC  approved? 

b.  Has  the  COIC  been  coordinated  with  the  MATDEV,  OPTEC  OEC, 
CASCOM,  and  HQ  TRADOC  TRASSO? 

c.  Have  all  concurred  with  the  COIC?  (If  "NO,"  strong 
rationale  must  be  provided  for  TRADOC  Proponent/DCSCD 
consideration. ) 

4.  COIC  Review  and  Approval  -  Sustaining  Base  and  Strategic 
MAISRC  Systems. 

a.  Are  the  MNS  and  ORD  approved? 

b.  Has  the  COIC  been  coordinated  with  and  concurred  in  by  CG 
OPTEC,  PM,  and  appropriate  DA  staff  elements? 

c.  Is  FP  COIC  briefing  ready  for  presentation  to  VDISC4? 
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COIC  DEVELOPMENT  SAMPLE 

THE  SITUATION: 

System  -  Communications  system  including  radio  set  (component 
of  the  user  system) #  and  net  control  station  (NCS)  with 
generator,  vehicle  and  crew. 

Need  -  High  speed,  secure  and  nonsecure,  jam  resistant  data 
communications  for  automated  systems . 

Mission  -  Deploy  to  theater  of  operations,  set  up,  initialize 
net,  provide  continuous  communications  support,  and  relocate 
components  (frequently)  to  survive. 

Deployment  -  Light  forces  divisions  through  battalion  command 
posts  and  key  operational  units. 

Employment  - 

-  Combined  and  joint  operations  control 

-  Division  systems  control  manages  net 

-  NCS  support  (dedicated  team  with  vehicle) 

-  Radio  set  support  (standard  logistics) 

Acquisition  Strategy  - 

-  Developmental  system  (NCS  and  radio  set) 

-  Uses  standard  truck,  shelter,  and  generator 

-  ORD  and  Milestone  II  approaches  approved 

-  MS  IIIA  (LRIP  based  on  technical  and  user  tests) 

-  MS  III  (full  production  based  on  technical  and  lOT) 

ORD  Requirements  Emphasis  -  v 

-  Connectivity  between  users  (communications  link  exists) 

-  Continuity  of  operations  during  movement  and  maintenance 

-  NCS  set  up,  tear  down,  and  net  initialization  times 

-  Aerial  deployment  for  NCS  (radio  certified  with  user) 

-  Allied  and  combined  operations  interoperability 

-  RAM  for  NCS  and  radio  set 

ORD  Requirements  -  .  .  * 

1  User  connectivity  90%  of  the  time  in  a  benign  environment 

2  User  connectivity  80%  of  the  time  in  an  EW  environment 

3  User  throughput  (messages/hour)  identified  by  the  user 
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4  User  speed  of  service  requirement  identified  by  the  user 
(not  more  than  a  factor  of  3  degradation  in  an  EW  environment for 
priority  messages) 

5  Continuity  of  net  operations  (NCS/radios)  during  movement 
and  maintenance 

6  NCS  roll-on/off  transportability  via  C-130 

7  NCS  certified  for  air  drop  and  Low  Altitude  Parachute 
Extraction  System  (LAPES)  deployment 

8  NCS  set  up  (first  radio  in  net)  within  45  minutes 

9  NCS  tear  down  and  depart  site  within  45  minutes 

10  High  Altitude  Electromagnetic  Pulse  (HAEMP)  and  Nuclear, 
Biological  and  Chemical  Contamination  (NBCC)  survivable 

11  Employed  in  hot,  basic,  and  cold  climates 

12  Communications  interface  with  allied  and  other  service 
communications  systems  used  with  automated  control  systems 

13  School  NCS  training  will  include  training  device  (one 
trainer  station  and  four  (4)  student  stations);  unit  sustainment 
training  will  be  supported  by  an  exportable  training  package 

14  Reliability,  Availability,  and  Maintainability  (RAM):  NCS 

A  .9,  Mean  Time  Between  Operational  Mission  Failure  (MTBOMF) 
300HR,  and  Maintenance  Ratio  (MR)  0.002;  Radio  Set  A  .95,  MTBOMF 
300HR,  and  MR  0.0005  ° 

Specification  Requirement  -  90%  throughput  success  and  90% 
speed  of  service  success,  given  user  connectivity  exists 

Operational  Mode  Summarv/Mission  Profile  fOMS/MP^  -  NCS  set  up 
within  45  minutes,  operate  for  2  hours,  tear  down  within  45 
minutes,  movement  1  hour,  24  hour/day  operations;  Radio  Set  lAW 
user  system  OMS/MP 

Approved  COI  for  Another  Communications  System  - 

-  Three  Issues  --  Does/Can  it 

—  Provide  secure  voice  and  data  communications  meeting 

user  need 

—  Deploy  from  garrison  to  field  and  operate  lAW  OMS/MP 
—  System  with  logistics  sustain  combat  operations 

-  Key  criteria  -- 

—  Probability  of  a  message  being  sent  and  received  in 
benign  and  Electronic  Warfare  (EW)  environments 

—  Movement  to  field  site  in  a  single  lift 
—  Set  up  and  tear  down  times 
—  Sustained  combat  operations  for  30  days 
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other  Considerations  -  , 

-  Technical  test  to  verify  technical  characteristics 

-  DIA  approved  threat  package  and  scenario  to  be  used  in 

Initial  Operational  Test  (lOT) 

-  lOT  to  test  total  operational  system 

-  Doctrine  and  Organization  Test  Support  Package  (TSP)  to 

be  used  for  employment  in  lOT 

-  COIC  Guidance  -  sustainment  COIC  for  a  control  system 
should  address  training  maintaining  proficiency  in  the  unit  and 
logistics  sustaining  combat  operations  for  a  period  of  time 

-  Approved  COIC  for  Another  System  Included: 
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COIC  Development  Sample  Continued 


A  SOLUTION! 


Critical  Operational  Issues  and  Criteria  (COIC) 
for  the  AN/GRC-986 ( V)  Communications  System 
for  Test  and  Evaluation  Master  Plan  (TEMP)  Supporting 

Milestone  II 

1.0  Issue;  Does  the  AN/GRC-986 (V)  system  provide  high  speed, 
secure  and  nonsecure,  jam  resistant  data  communications  for  light 
forces  automated  control  systems? 

1.1  Scope:  This  issue  examines  the  capability  of  the 
AN/GRC-986 (V)  to  provide  high  speed,  secure  and  nonsecure,  jam 
resistant  communications  support  for  light  forces,  to  include 
combined  and  joint  operations.  A  division  slice  will  be  played 
with  radios  for  allies  and  other  services  control  systems  in  a 
net.  Communications  measure  of  performance  to  be  examined  will 
be  percentage  of  message  traffic  passed.  The  AN/GRC-986 (V)  will 
be  operated  and  maintained  by  Qualified  soldiers  in  accordance 
with  the  Operational  Mode  Summary/Mission  Profile  (OMS/MP) . 
Continuity  of  operations  during  movement  and  maintenance  will 
occur  as  a  normal  part  of  operations.  Employment  will  be  in 
accordance  with  the  Doctrinal  and  Organizational  Test  Support 
Package  (TSP) .  MOPP  IV  level  operations  will  be  simulated. 

1 . 2  Criteria . 

1.2.1  The  AN/GRC-986 (V)  will  pass  at  least  73%  of  the  user 
required  priority  message  traffic  to  the  correct  addressee  within 
the  user  specified  speed  of  service  (SOS)  (see  note  3)  in  a 
benign  and  at  least  65%  or  priority  messages  with  no  more  than  a 
factor  of  3  degradation  in  SOS  in  a  threat  EW  environment. 

1.2.2  Given  compatible  automated  control  systems,  the 
AN/GRC-986 (V)  will  interface  with  allied  and  other  service 
systems  (see  note  4  for  systems)  to  exchange  data. 

1*3  Rationale .  The  AN/GRC-986 (V)  mission  effectiveness  is  its 
capability  to  deliver  information  to  the  correct  addressee  in 
time  to  take  necessary  action.  During  combined  and  joint 
operations,  other  services  or  allies  are  part  of  the  mission  and 
all  must  exchange  required  data. 


1.3.1  Criterion  1.2.1  was  derived  from  ORD  requirements 
paragraphs  1,  2,  3,  and  4  (connectivity  in  benign  and  EW 
environments,  throughput,  and  SOS)  and  specification  requirements 
for  90%  throughput  and  90%  SOS.  Benign  percentage  =  .9  X  .9  X  .9 
X  100  =  73%.  EW  percentage  =  .8  X  .9  X  .9  X  100  =  .65. 

1.3.2  Criterion  1.2.2  comes  from  ORD  requirement  paragraph  12. 

2.0'  Issue;  Can  the  AN/GRC-986 (V)  be  deployed  from  garrison  to  a 
field  site  and  operate  in  accordance  with  the  OMS/MP? 

2.1  Scope;  This  issue  examines  the  deployability  of  the 
AN/GRC-986 (V)  as  a  total  system,  i.e.,  shelter/truck  mounted 
radio  set  and  NCS  with  organic  generator.  Specific 

modes /techniques  of  deployability  addressed  will  be 
roll-on/roll-off  and  aerial  delivery  via  Low  Velocity  Air  Drop 
(LVAD)  and  LAPES  from  C-130  aircraft.  The  crew  will  be  deployed 
by  separate  aircraft.  Additionally,  data  will  be  collected  in 
benign  and  NBC  (MOPP  IV)  environments  on  the  time  required  to 
prepare  the  system  (set  up)  for  operation  following 
crew/equipment  link-up  and/or  arrival  at  the  operations  site,  and 
to  prepare  the  system  (tear  down)  for  survivability  moves. 

2.2  Criteria; 

2.2.1  The  AN/GRC-986 (V)  net  control  station  (NCS)  must  be 
certified  for  the  following  transport  and  deployment  methods; 

a.  Roll-on  and  roll-off  transport  by  C-130. 

b.  LVAD  (air  drop)  and  LAPES  delivery. 

2.2.2  The  NCS  crew  must  set  up  and  have  the  first  radio  in  the 
net  within  45  minutes  90%  of  the  time  (time  starts  upon  arrival 
on  site).  When  dressed  in  MOPP  IV,  60  minutes  is  allowed. 

2.2.3  The  NCS  crew  will  tear  down  and  depart  site  with  median 
time  less  than  45  minutes  after  receipt  of  the  move  order.  A 
median  time  of  60  minutes  is  allowed  when  dressed  in  MOPP  IV. 

2.3  Rationale;  While  the  AN/GRC-986 (V)  NCS  will  be  transported 
via  all  modes,  aerial  deployability  is  most  critical  to  light 
units.  The  NCS  must  be  like  deployable  to  the  users  it  supports. 
The  NCS  must  move  to  survive  during  combat. 
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2.3.1  Criterion  2.2.1  is  derived  from  ORD  paragraphs  6  and  7. 

2.3.2  Criterion  2.2.2  comes  from  ORD  requirement  paragraph  8. 
Applying  a  90  percent  factor  recognizes  the  possibility  of 
shortfalls  under  operational  conditions.  Set  up  is  considered 
more  time  sensitive  than  tear  down.  An  allowance  of  15 
additional  minutes  is  made  for  MOPP  IV  degradation. 

2.3.3  Criterion  2.2.3  is  based  on  ORD  requirement  paragraph  9, 
with  similar  considerations  to  those  for  criteria  2.2.2.  Median 
time  is  considered  realistic  for  tear  down. 

3.0  Issue;  Can  AN/GRC-986 ( V)  equipped  units  achieve  training 
proficiency  in  garrison  and  provide  a  wartime  readiness 
capability  for  sustained  combat  operations? 

3 . 1  Scope; 

3.1.1  This  issue  examines  sustainment  training  provided  to  NCS 
crews.  The  unit  training  device,  training  publications  and 
literature,  and  methods  of  instruction  included  in  the  program  of 
instruction  will  be  addressed.  Training  adequacy  will  be 
examined  in  terms  of  operator  proficiency  in  performing  critical 
tasks  required  to  effectively  employ  the  AN/GRC-986 (V)  (the 
critical  tasks  and  standards  to  be  met  will  be  Identified  in  the 
training  TSP).  Questionnaires  and  structured  Interviews  with  the 
test  participants,  instructors,  and  test  directorate  personnel 
regarding  the  adequacy  of  training,  the  training  device,  training 
materials,  and  operator  acceptability  of  training  manuals  in 
accordance  with  AR  310-3  will  be  conducted.  Also  addressed  will 
be  correctness,  applicability,  format,  degree  of  detail,  and  ease 
of  use  of  publications. 

3.1.2  This  issue  also  encompasses  an  evaluation  of  the 
maintenance  concept,  system  support  package  (SSP),  and  PLL/ASL 
under  operational  conditions.  To  be  examined  are  the  dedicated 
NCS  maintenance  team,  and  logistics  support  hardware  and  software 
needed  to  support  the  system.  Hardware  includes  tools  and  test 
equipment.  Software  includes  technical  manuals,  repair  parts  and 
special  tools  listings,  the  maintenance  allocation  chart  (MAC), 
and  parts  allocation  tables.  Operational  conditions  will  include 
movement  to  enhance  survivability. 
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3 . 2  Criteria; 

3.2.1  The  AN/GRC-986(V)  NCS  crews  will  be  able  to  practice  and 
perform  crew  drills  in  garrison.  95%  of  the  representative 
soldiers  must  be  capable  of  performing  all  critical  tasks,  for 
their  respective  MOS,  to  the  assigned  training  standard. 

3.2.2  The  dedicated  NCS  maintenance  teams  (1  per  NCS),  with 
allotted  tools,  test  equipment,  and  repair  parts,  will  sustain  a 
division  operation  for  a  period  of  30  days  without  negative 
impact  on  continuity  of  operations. 

3.3  Rationale:  Units  will  come  to  combat  "as  is;"  therefore, 
they  must  maintain  proficiency  during  peacetime  and  be  capable  of 
sustaining  operations  until  the  logistics  system  catches  up. 

3.3.1  Criterion  3.2.1  is  based  on  ORD  requirement  paragraph  13, 
which  plans  for  an  exportable  packet  for  sustainment  training. 

3.3.2  Criterion  3.2.2  is  based  on  ORD  requirement  paragraph  14 
and  the  support  concept  of  providing  a  dedicated  maintenance 
teams  for  the  NCS.  The  30  day  sustainment  factor  is  minimum 
essential  to  allow  the  logistics  system  to  catch  up. 


Note  1;  Criteria  are  total  system  measures.  As  such,  they 
inherently  cover  hardware,  software,  personnel,  doctrine, 
organization,  and  training.  System  individual  characteristics  of 
operational  capability,  survivability,  RAM,  organization, 
doctrine,  tactics,  logistics  support,  training,  and  MANPRINT 
(which  includes  the  domains  of  manpower,  personnel,  training, 
human  factors  engineering,  system  safety,  and  health  hazards) 
related  to  these  criteria  will  be  provided  by  the  operational 
independent  evaluator  in  the  operational  test  and  evaluation 
plan. 

Note  2:  Criteria  are  not  provided  as  automatic  (default) 
pass/fail  measures.  Rather,  they  represent  estimates  of 
performance  for  which  a  breach  would  require  a  careful  senior 
level  management  reassessment  of  cost  effectiveness  and  program 
options  during  the  program  milestone  decision  review. 
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Note  3:  This  note  would  contain  a  definition  of  user  specified 
speed  of  service  (SOS). 

Note  4:  This  note  would  contain  a  listing  of  Allied  and  other 
service  systems  with  which  the  AN/GRC-986(V)  is  required  to  be 
interoperable  for  data  exchange. 
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Chapter  1 

Developmental  Test  &  Evaluation  in  the  Acquisition  Process 


Section  I 
General 


1-1.  Developmental  Test  and  Evaluation  (DT&E) 

DT&E  is  conducted  throughout  the  acquisition  process  to  assist  in 
the  engineering  design  and  development  of  a  system  and  to  verify 
that  developmental  performance  specifications  have  been  met. 

a.  Developmental  testing  is  a  generic  term  encompassing 
engineering-type  testing,  generally  requiring  instrumentation  and 
measurements,  which  is  accomplished  by  engineers,  technicians, 
and/or  soldier  operator-maintainer  test  personnel.  It  includes 
technical  feasibility  testing,  engineering  development  testing, 
software  development  testing,  production  and  post-production 
testing,  post-deployment  software  support  testing,  and  testing 
associated  with  vulnerability/ lethality  assessments. 

b.  Developmental  tests  are  designed  to  subject  the  system  or 
its  components,  both  hardware  and  software,  to  stress  levels 
commen-  surate  with  those  to  which  the  mature  system  will  be 
subjected  in  all  operating  environments.  If  required, 
developmental  tests  may  subject  the  system  to  stress  levels  which 
will  estimate  the  outer  limits  of  the  operational  envelope. 
Developmental  testing  determines  the  system  safety,  technical 
performance,  MANPRINT,  human  factors  performance,  and  the 
integrity  of  the  equipment.  All  government  developmental  testing 
and  associated  production  testing  on  materiel  systems  are 
executed  by  U.S.  Army  Test  and  Evaluation  Command  (TECOM)  unless 
otherwise  designated  in  the  Test  and  Evaluation  Master  Plan 
(TEMP) .  Developmental  testing  of  information  systems  is 
conducted  at  U.S.  Army  Information  Systems  Command  (ISC) 
facilities  or  at  TECOM  facilities,  and  the  responsibility  for 
developmental  testing  of  strategic  defense  systems  is  U.S.  Army 
Strategic  Defense  Command  (SDC) .  Section  XXII,  Chapter  3, 
contains  procedures  for  requesting  test  and  test  support  services 
from  TECOM. 

c.  When  programs  experience  technical  or  operational  problems, 
developmental  testing  and  evaluation  provides  a  valuable  service 
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by  helping  to  find  problems  and  verify  fixes  before  they  become 
show-stoppers.  A  concerted  effort  is  required  by  the  tester  and 
developer  to  mature  the  equipment  technically  and  properly  test 
it  before  transitioning  it  for  operational  testing  or  production 
processes. 

1-2.  Developmental  Independent  Evaluation/Assessment 
The  developmental  independent  evaluator /assessor  assists  in  the 
engineering  design  and  development  and,  through  the  continuous 
evaluation  process,  determines  the  degree  to  which  the  technical 
parameters  of  the  system  have  been  achieved.  The  evaluator/ 
assessor  optimizes  the  use  of  data  obtained  from  models, 
simulations,  and  test  beds,  as  well  as  tests  conducted  on 
prototypes  or  full-scale  development  models  of  the  system.  The 
developmental  independent  evaluator  is  U.S.  Army  Materiel  Systems 
Analysis  Activity  (AMSAA)  for  major  materiel  systems,  and  TECOM 
is  the  developmental  assessor  for  nonmajor  materiel  systems. 

Also,  SDC  has  responsibility  for  the  developmental  independent 
evaluation  of  special  SDC  programs,  and  the  ISC  is  responsible 
for  developmental  evaluation  of  assigned  information  systems. 


Section  II 

Major  DT&E  Actions  during  the  Acquisition  Cycle:  Materiel 
Systems 


1-3.  DT&E  Planning 

a.  The  materiel  developer,  in  coordination  with  Test 
Integration  Working  Group  (TIWG)  members,  structures  T&E  programs 
concurrently  with  the  acquisition  strategy.  Consideration  must 
be  given  to  T&E  over  the  system's  entire  life  cycle.  Program 
planning  documents  (Section  XXII)  are  a  source  of  information  to 
assist  the  materiel  developer  and  the  developmental  tester  to 
identify  future  test  resource  requirements  (e.g.,  personnel, 
funds,  facilities,  instrumentation) . 

b.  DT&E  sufficient  to  address  the  critical  technical 
parameters  must  be  done  before  each  major  decision  point  to 
reduce  acquisition  risks  and  to  estimate  the  capability  of  the 
system  to  meet  the  technical  requirements.  Developmental  test 
programs  will  be  structured  to  provide  sufficient  data  to  allow 
evaluation  of  issues  regarding,  but  not  limited  to, 
effectiveness,  safety,  performance,  PAM,  and  MANPRINT 
considerations.  The  developmental  independent 
evaluators/assesscrs  provide  the  milestone  decision  authority 
with  information  that  addresses  the  critical  technical  parameters 
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specifying  those  which  have  been  designated  as  exit  criteria. 

Exit  criteria  are  the  specific  minimum  requirements  that  must  be 
satisfactorily  demonstrated  before  the  program  can  progress  into 
the  next  acquisition  phase  (DODI  5000.2,  Part  11,  Section  A). 

c.  Developmental  testing  is  planned  and  conducted  to  take  full 
advantage  of  the  existing  investment  in  DOD  ranges  and  other  test 
facilities,  whenever  practical.  Agencies  with  requirements  for 
developmental,  production,  or  post-production  testing  of  military 
materiel  use  DOD  Major  Range  and  Test  Facility  Base  (MRTFB) 
activities  and  other  DA  test  facilities  instead  of  establishing 
in-house  capabilities  or  contracting  for  testing  services. 
Exceptions  are  justified  in  the  TEMP. 

(1)  The  DA  MRTFB  is  an  aggregation  of  test  activities, 
facilities,  ranges,  and  equipment  designed  to  provide  the  Army 
with  the  best  overall  military  T&E  capability.  The  MRTFB  is 
operated  and  managed  under  uniform  reimbursement  policy.  DOD 
test  customers  utilizing  the  MRTFB  are  required  to  pay  only  those 
costs  that  are  directly  identified  to  the  test.  The  indirect  or 
overhead  costs  are  funded  by  the  MRTFB  activity's  parent  command 
(see  AR  70-69  and  DODD  3200.11). 

(2)  The  MRTFB  and  other  test  facilities  are  capital 
investments  designed  to  provide  comprehensive  testing 
capabilities  that  support  all  materiel  acquisition  programs. 

These  facilities  have  unique  capabilities  and  expertise  and  offer 
significant  cost  benefits  to  DA  test  customers. 

(3)  DA  MRTFB  activities  are:  Yuma  Proving  Ground,  Dugway 
Proving  Ground,  U.S.  Army  Combat  Systems  Test  Activity  (located 
at  Aberdeen  Proving  Ground),  White  Sands  Missile  Range,  U.S.  Army 
Electronic  Proving  Ground  (located  at  Fort  Huachuca) ,  and 
Kwajalein  Atoll.  Section  XXI,  Chapter  3,  contains  a  brief 
description  of  the  DA  test  capabilities,  including  MRTFB 
activities.  DOD  3200. 11-D  provides  a  summary  of  capabilities  of 
the  DOD  MRTFB. 

d.  As  requirements  are  being  generated,  close  interaction  is 
maintained  with  ongoing  technology  development  programs  to  ensure 
focus  on  critical  military  needs.  Prior  to  establishment  of  a 
program  office  or  approval  of  a  TEMP,  Research  Efforts/Tests, 
including  Advanced  Technology  Transition  Demonstrations  (ATTD) 
and  Proof  of  Principle  (POP)  demonstrations,  may  be  used  to 
examine  the  feasibility  of  alternative  technologies  and  to 
expedite  technology  transition  from  the  laboratory  to  operational 
use. 
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1-4.  Concept  Exploration  and  Definition  (Phase  0) 

Milestone  (MS)  0  approves  the  conduct  of  concept  studies  to 
examine  potential  solutions  in  response  to  an  identified  mission 
need  and  begins  Phase  0.  During  this  phase,  the  Acquisition 
Strategy  is  developed,  the  Operational  Requirements  Document 
(ORD)  is  written  for  each  concept  being  studied,  a  TIWG  is 
formed,  and  a  draft  TEMP  is  developed.  The  developmental  tester 
and  the  developmental  evaluator/  assessor,  as  members  of  the 
TIWG,  provide  input  to  the  TEMP  (see  Part  Two,  TEMP  Guidelines). 

a.  As  the  ORD  and  the  TEMP  are  being  staffed,  the 
developmental  independent  evaluator/assessor  begins  preparation 
of  the  Independent  Evaluation/ Assessment  Plan  (lEP/IAP) .  The 
lEP/IAP  is  coordinated  with  the  TIWG  members. 

b.  As  issues  identified  in  the  lEP/IAP  are  analyzed  and 
satisfied  during  the  continuous  evaluation  effort,  they  are 
retained  and  annotated  in  the  lEP/IAP,  but  are  not  included  in 
future  evaluation  efforts  unless  program  changes  indicate  the 
issue  should  be  reevaluated.  Those  requiring  more  evaluation  are 
addressed  in  follow-on  evaluation  actions  in  subsequent 
acquisition  phases.  New  evaluation  issues  are  added,  as 
appropriate,  and  data  source  matrices  and  test  documents  are 
revised  accordingly. 

c.  Technical  feasibility  tests  (TFT)  are  conducted  to  explore 
materiel  concepts  and  refine  evaluation  issues.  The  hardware 
configuration  will  generally  be  breadboards,  components, 
subsystems,  brassboards,  and/or  experimental  prototypes.  TFTs 
assist  in  determining  safety  and  establishing  system  performance 
specifications  and  feasibility. 

d.  The  first  iteration  of  the  TEMP  is  approved  at  MS  I,  which 
initiates  the  development  program  and  approves  proceeding  into 
the  demonstration  and  validation  phase. 

1-5.  Demonstration/Validation  Phase  (Phase  I) 

During  this  phase,  the  TIWG  updates  the  Acquisition  Strategy, 
develops  integrated  plans  for  T&E,  and  the  TEMP  is  updated, 
coordinated,  and  approved  by  the  decision  authority  at  MS  II. 

Once  the  TEMP  is  approved,  deletion  of  tests  requires  a  waiver  by 
the  TEMP  approval  authority. 

a.  The  developmental  independent  evaluator/assessor ,  as  a 
member  of  the  TIWG,  integrates  evaluation  requirements.  To 
optimize  evaluation,  both  contractor  and  government  data  are 
shared  by  the  user,  developer,  and  independent 

evaluator /assessor.  The  materiel  developer  is  responsible  for 
providing  all  data  to  the  independent  developmental 
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evaluator/assessor . 


b.  The  TIWG  members  assist  in  the  development  of  the 
Integrated  Test  Program  Schedule  (ITPS)  which  identifies  the  type 
of  tests  to  be  conducted  and  the  number,  scope,  and  schedule  of 
test  activities  for  the  system  throughout  its  acquisition 
process . 


c.  Testing,  modeling,  and  simulations  are  conducted  as  planned 
in  the  TEMP.  Engineering  Development  Tests  (EDT)  are  conducted 
during  this  phase  to  provide  data  on  safety,  the  achievability  of 
critical  technical  parameters,  refinement  and  ruggedization  of 
hardware  configurations,  and  determination  of  technical  risks. 

The  EDT  provides  data  on  the  compatibility  and  inter-  operability 
with  existing  or  planned  equipment  and  systems  and  the  system 
effects  caused  by  natural  and  induced  environmental  conditions 
during  the  development  phase. 


d.  The  MS  II  decision  approves  the  development  of  the  item  and 
initiates  Phase  II. 


1-6.  Engineering  and  Manufacturing  Development  phase  (Phase  II) 


a.  During  the  phase  leading  up  to  the  MS  III  production 
decision,  developmental  testing  supports  the  hardware  (or  sys  em) 
and  associated  software  design  through  a  test-analyze-f ix-test 
approach  performed  at  the  component,  subsystem,  and  system  leve  , 
identifies  the  preferred  technical  approach,  including  the 
identification  of  technical  risk  and  feasible  solutions;  examines 
the  operational  aspects  of  support  requirements  and  the  selected 
alternative  technical  approaches;  provides  preliminary  data  on 
the  potential  operational  effectiveness  and  suitability  of 
candidate  systems;  supports  the  materiel  improvement  process; 
identifies  design  risk;  establishes  contractual  compliance 
including  component  qualifications;  makes  preliminary  assessment 
of  MANPRINT  requirements;  provides  data  for  required  test 
readiness  reviews;  and  evaluates  the  technical  parameters.  (See 
Seven  for  details  on  software  T&E  of  materiel  system 
computer  resources.) 


b  A  Production  Proveout  Test  (PPT)  may  be  conducted,  prior  to 
production  testing,  to  determine  the  most  appropriate  design 
alternative.  During  this  phase,  Pre-production  Qualification 
Test  (PPQT)  is  conducted  to  ensure  design  integrity  over  the 
specified  operational  and  environmental  range.  A  Live  Fire  T&E 
may  be  required  during  this  phase  on  some  systems  (see  Part  Six) . 
(See  Section  XX,  Chapter  3,  for  other  developmental  tests  that 
may  be  required.) 
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c.  A  favorable  MS  III  decision  is  a  commitment  to  produce  and 
support  the  system. 

1-7.  Production  and  deployment  phase  (Phase  III) 

Developmental  testing  during  this  phase  verifies  that 
requirements  specified  in  the  Technical  Data  Package  and 
production  contracts  are  met  and  provides  test  data  for  materiel 
release  action.  It  determines  if  the  production  item  fulfills 
the  users'  requirement.  It  is  the  soldier's  guarantee  that 
performance  and  quality  have  not  been  lost  in  the  transition  from 
development  to  production. 

a.  For  items  that  are  to  be  procured  by  the  Defense  Logistics 
Agency,  the  Army  is  responsible  for  certifying  that  the 
production  item  conforms  to  the  specifications  and  that 
performance  is  not  lost  in  the  production  process.  The  data 
required  for  this  certification  are  derived  from  the  government 
Production  Qualification  Test  (PQT)  of  the  first  contract. 

b.  Testing  during  the  production  phase  of  the  materiel  life 
cycle  is  a  logical  flow-down  of  pre-MS  III  tests  and  includes  the 
testing  necessary  to  verify  that  requirements  specified  in 
technical  data  packages  and  production  contracts  for  hardware  or 
software  are  met.  Production  testing  also  provides  a  baseline 
for  follow-on  post  production  testing.  Production  Qualification 
Tests  (PQT)  and/or  First  Article  Tests  (FAT)  are  conducted  to 
verify  the  production  item  meets  contract  requirements.  A 
Follow-on  Production  Test  (FPT)  may  be  conducted  to  verify  the 
adequacy  of  corrective  actions  indicated  by  the  PQT.  Other 
production  testing  includes  Comparison  Tests  (CPT)  and  Quality 
Conformance  Inspections.  (See  Section  XX,  Chapter  3,  for 
detailed  descriptions  of  all  production  tests.) 

c.  Planning,  programming,  and  budgeting  for  testing  during 
production  will  begin  early  in  the  development  cycle.  Funding 
will  be  as  prescribed  in  AR  73-1  and  AR  37-100.  In  general, 
production  items  are  procured  with  Procurement,  Army  (PA)  funds, 
and  the  procurement  of  repair  parts  are  Operations  and 
Maintenance,  Army  (OMA)  funded;  costs  of  conducting  tests  are 
similarly  funded. 

d.  Provisions  will  be  made  in  the  Long-Range  Army  Materiel 
Requirements  Plan  (LRAMRP)  for  test  items,  facilities, 
instrumentation,  and  resources  to  support  quality  assurance 
testing  during  production. 

e.  Criteria  for  production  testing  will  be  prescribed  in  the 
appropriate  Technical  Data  Package  based  upon  the  performance 
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demonstrated  during  development  or  in  the  contract  performance 
specifications.  Description  of  the  test,  methods  of  analysis, 
and  pass/ fail  criteria  will  be  included.  The  number  of  items  to 
be  tested  and  the  duration  of  tests  will  be  based  upon  sound 
engineering  practices  in  consideration  of  costs,  schedules,  item 
complexity,  known  problem  areas,  risks  (confidence  levels), 
other  factors.  Advantage  will  be  taken  of  prior  test  data  and 
analytically  derived  design  data  in  developing  the  test  and 
sampling  plan.  Acceptable  quality  levels  (AQLs)  will  not  be  used 
in  association  with  acceptance  of  materiel  from  a  production 
contract . 

f.  The  total  system  is  tested  during  PQT.  When  individual 
components  and  subsystems  are  tested  separately ,  such  testing  in 
itself  will  not  be  considered  as  meeting  total  system  test 
requirements. 

g.  Proponent  activities  will  establish  procedures  to  assure 
the  timely  planning,  testing,  reporting  and  resolution  of 
deficiencies  on  newly  procured  materiel,  and  ensure  that 
developmental  test  requirements  are  identified  to  allow 
appropriate  flexibility  regarding  tests,  such  as: 

(1)  Tailoring  of  sample  sizes  to  meet  specific  contract 
requirements. 

(2)  Termination  during  early  testing  if  performance  is  so 
poor  that  retesting  will  be  required  regardless  of  the  results  of 
the  remaining  portion  of  the  tests. 

(3)  Reduction,  elimination,  or  early  termination  of  certain 
tests  when  there  is  sufficient  evidence  that  requirements  have 
been  demonstrated  with  high  confidence. 

1-8.  Post-production  testing 

Post— production  testing  is  conducted  to  assure  that  materiel 
which  is  stored,  reworked,  repaired,  renovated,  or  rebuilt  after 
initial  issue  conforms  to  specified  quality,  reliability,  safety, 
MANPRINT,  and  technical  and  operational  performance  standards. 
(Post-production  tests  types  are  detailed  in  Section  XX,  Chapter 

3-) 

a.  Post— production  testing  is  a  follow-on  to  production 
testing  and  includes  those  surveillance  and  reconditioning  tests 
required  to  measure  the  ability  of  materiel  in  the  field,  in 
storage,  and  after  maintenance  actions  (to  include  repair, 
rebuild,  retrofit,  overhaul,  and  modification)  to  meet  user 
requirements . 
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b.  After  fielding,  materiel  continues  to  be  tested,  to  be  sure 
that  it  is  holding  up  in  storage  and  is  fully  functional, 
reliable,  and  operable.  Testing  is  done  under  conditions  that 
approximate  as  closely  as  possible  what  would  be  experienced 
under  actual  field  conditions. 

c.  Planning,  programming,  and  budgeting  for  testing  during  the 
post-  production  phase  will  begin  early  in  the  development  cycle. 
Funding  will  be  as  prescribed  in  AR  37-100.  In  general,  post¬ 
production  testing,  to  include  materiel  and  conduct  of  tests  will 
be  DMA  funded. 

d.  Criteria  for  surveillance  testing  will  be  prescribed  in  the 
appropriate  Bulletins  (TBs) ,  Manuals  (TMs) ,  Storage 
Serviceability  Standards,  and  Surveillance  Test  Program  Plans. 

e.  Criteria  for  reconditioning  testing,  including  pilot 
reconditioning  tests,  initial  reconditioning  (first  article) 
tests,  control  (comparison)  tests,  acceptance  tests,  total 
systems  tests,  and  baseline  evaluation  tests  at  depot  or 
contractor  facilities,  will  be  incorporated  in  Depot  Maintenance 
Work  Requirements  (DMWRs) ,  Modification  Work  Orders  (MWOs) ,  TMs, 
TBs,  and  contracts. 

f.  Test  criteria  will  be  based  on  performance  demonstrated 
during  development  and  production.  The  number  of  items  to  be 
tested  and  the  duration  of  tests  will  be  based  upon  sound 
engineering  practices  considering  schedules,  costs,  item 
complexity,  known  problem  areas,  risks  (confidence  levels) , 
system  and  software  changes  made,  and  other  factors.  Advantage 
will  be  taken  of  prior  test  data  and  analytically  derived  design 
data  in  developing  the  test  and  sampling  plan.  Existing  test 
facilities  will  be  used  rather  than  building  new  government  or 
contractor  facilities. 


Section  III 

Major  DT&E  Actions  during  the  Acquisition  Cycle:  Information 
Systems 


1-9.  Information  System  Categories 

The  two  categories  of  Information  Systems  (theater/tactical  and 
nontheater/tactical)  are  defined  in  AR  73-1  (para  3-2) . 

a.  Theater /tactical  information  systems  follow  the  same  phases 
and  milestones  as  materiel  systems. 
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(1)  Developmental  testing  of  information  systems  includes 
Software  Development  Tests,  Software  Qualification  Tests,  and 
Post-Deployment  Software  Support  tests.  Software  Development 
Tests  are  conducted  by  the  developer  on  programs  and  modules; 
qualification  tests  are  conducted  during  Phase  II  by  the 
government  developmental  tester  at  the  system  level  and  include  a 
developmental  independent  evaluation/assessment;  and  post¬ 
deployment  tests  consist  primarily  of  modifications  and 
maintenance  of  software.  (Section  XX,  Chapter  3  defines  the 
software  test  types  that  are  conducted  by  the  government 
developmental  tester.) 

(2)  System-level  developmental  testing  is  conducted  at 
stress  levels  representative  of  data  volumes  expected  to  be 
encountered  under  the  most  extreme  circumstances  (e.g., 
deployment  surge,  wartime  operation  with  full  force  structure 
participation,  and  year-end  closeout  processing).  Developmental 
testing  will  be  structured  to  estimate  the  outer  limit  of  the 
system's  operational  envelope. 

b.  Nontheater /tactical  information  systems  follow  a  slightly 
different  life  cycle,  in  that  the  phases  do  not  correspond 
directly  to  the  materiel  system  life  cycle  phases.  The  life 
cycle  phases  are  compared  in  Figure  3-1,  AR  73-1.  All 
developmental  testing  of  nontheater /tactical  informations  systems 
is  conducted  by  the  developer  in  conjunction  with  the  user. 

1-10.  Software  T&E 

For  more  detailed  information  regarding  software  T&E,  see  Part 
Seven. 


Section  IV 

Nondevelopmental  Item  T&E 


1-11.  General 

Nondevelopmental  items  (NDI)  provide  a  preferred  alternative  if 
the  market  surveillance  reveals  that  items  are  available  which 
have  a  high  probability  of  meeting  the  users'  requirements.  NDI 
feasibility  may  surface  before  preparation  of  the  Mission  Need 
Statement  (MNS)  or  may  be  identified  during  the  market 
investigation.  This  is  based  upon  continuous  market 
surveillance,  front-end  analysis,  responses  to  Mission  Area 
Analysis  deficiencies,  and  the  proposed  solution  in  the  Materiel 
Acquisition  Decision  Process  (MADP) .  The  market  investigation 
becomes  much  more  important  as  a  data  source  for  NDI  systems  and 
often  is  the  only  source  prior  to  the  MS  I/III  decision  review. 
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T&E  requirements  to  support  an  NDI  acquisition  approach  do  not 
differ  appreciably  from  the  T&E  requirements  for  a  developmental 
program:  a  TIWG  must  still  be  formed;  a  TEMP  is  required;  test 

data  must  be  available;  and  a  developmental  evaluation/assessment 
must  be  performed  by  the  developmental  IE. 

1-12.  NDI  Acquisition 

The  foundation  of  an  acquisition  strategy  is  the  market 
investigation.  Prior  to  the  MS  I  decision,  both  the 
developmental  and  operational  evaluator  prepare  evaluation  plans 
which  are  used  to  guide  the  market  investigation.  These  contain 
system-specific  questions  (e.g.,  performance,  MANPRINT, 
operation,  design  features)  that  must  be  answered  during  the 
market  investigation  process.  The  answers  to  these  questions 
will  generally  require  the  evaluation/assessment  of  existing 
commercially  available  test  data,  technical  feasibility  test 
results,  or  user  experiment  data. 

1-13.  NDI  T&E  Master  Plan 

If  the  results  of  the  market  investigation  indicate  that  an  NDI 
solution  is  feasible,  the  TEMP  will  reflect  the  test  activities, 
if  any,  required  to  do  one  of  the  following: 

a.  Proceed  directly  to  a  combined  MS  I/III  decision  review 
which  makes  the  production  and  type  classification  decision;  or 

b.  Proceed  to  an  MS  I/II  decision  review.  In  this  instance 
the  requirement  for  a  Preproduction  Qualification  Test  (PPQT)  of 
the  NDI  candidates  will  be  initiated.  This  test  will  be 
conducted  to  determine  if  requirements  are  fully  satisfied.  The 
extent  of  modifications,  if  required,  is  one  factor  which 
determines  if  and  how  much  OT  is  necessary. 

1-14.  DT&E  for  NDI 

Developmental  testing  requirements  will  be  tailored  to  each 
specific  system.  DT&E  will  be  conducted,  at  a  minimum,  to  verify 
integration  and  interoperability  with  other  system  elements. 
Additional  T&E,  as  appropriate,  will  be  conducted  to  evaluate  and 
control  risk.  The  following  is  provided  as  general  guidance,  not 
rigid  requirements,  of  the  testing  activities  appropriate  for  the 
following  NDI  options: 

a.  Off-the-shelf  items  to  be  used  in  the  same  environment  for 
which  they  were  designed  (i.e.,  no  development  or  modification  of 
hardware  or  software  is  required)  will  normally  not  require 
developmental  testing  before  MS  I/II  or  I/III;  however.  Force 
Development  Tests  &  Experiments  and  TFTs  may  be  conducted  to 
support  the  MS  decision.  When  the  contract  is  awarded  to  a 
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contractor  who  has  not  previously  produced  an  acceptable  finished 
product  and  the  item  is  assessed  as  high  risk,  a  PQT  should  be 
required . 

b.  Those  off-the-shelf  items  which  require  some  modification 
of  hardware  or  operational  software  (e.g.,  militarization  or 
ruggedization)  will  require  TFT,  unless  the  decision  authority 
documents  that  further  testing  is  not  required.  PPQT  is  required 
if  feasibility  testing  results  in  the  necessity  for  fixes  to  the 
item.  Production  qualification  testing  (PQT)  is  required  to 
support  materiel  release. 

c.  An  R&D  effort  is  required  for  NDI  items  used  as  subsystems, 
modules,  or  components  which  contribute  to  a  materiel  solution. 
Systems  engineering,  software  modification,  and  testing  is 
required  to  ensure  the  total  system  meets  user  requirements  and 
is  producible  as  a  system.  TFT  is  required  in  a  military 
environment.  PPQT  of  the  complete  system  is  required.  Hardware/ 
computer  software  integration  tests  are  required  and  both  PQT  and 
OT  are  required. 

d.  Some  follow-on  testing  of  the  NDI  may  be  required  to  verify 
the  adequacy  of  corrective  actions  indicated  by  the  PQT. 

1-15.  OT&E  for  NDI 

See  Part  Five  for  OT&E  requirements  for  NDI. 


Section  V 
Live  Fire  T&E 


1-16.  General 

Through  a  series  of  amendments  to  Title  10,  United  States  Code, 
Congress  has  mandated  that  major  weapon  system  and  munition 
programs  (see  DODI  5000.2,  Part  8)  undergo  a  realistic  Live  Fire 
Test  and  Evaluation  (LFTE)  program.  LFTE  is  part  of  the 
developmental  testing  of  the  system's  vulnerability  and 
lethality.  The  scope  of  LFTE  should  build  upon  early 
developmental  tests  of  components  and  system  vulnerability  and 
lethality  modeling  and  should  be  reflected  in  the  TEMP.  See  Part 
Six  for  detailed  information  regarding  LFTE. 


Section  VI 

Clothing  and  Individual  Equipment  DT&E 
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1-17.  General 

The  acquisition  of  clothing  and  individual  equipment  (CIE)  is 
governed  by  AR  700-86.  The  overall  philosophy  is  very  similar  to 
the  process  described  in  AR  70-1  except  that  an  Army  Clothing  and 
Equipment  Board  and  a  Clothing  Advisory  Group  recommend  items  for 
approval  by  the  Vice  Chief  of  Staff,  Army. 

1-18.  Requirements  for  DT&E 

Testing  requirements  are  outlined  in  part  III  of  a  Statement  of 
Need  of  the  CIE.  Upon  procurement  of  a  CIE  item,  government 
initial  production  testing  should  be  conducted  to  certify  the 
specifications  so  that  future  procurements  and  Defense  Logistics 
Agency's  quality  control  are  effective.  T&E  management  documents 
for  the  acquisition  of  CIE  are  the  same  as  those  required  for  the 
acquisition  of  ACAT  III  and  ACAT  IV  systems  acquired  under  the 
auspices  of  AR  70-1,  i.e.,  TEMP,  lAP,  DTP,  TR,  and  lAR. 


Section  VII 

DT&E  in  Support  of  Type  Classification 


1-19.  General 

Type  classification  is  the  process  which  identifies  the  degree  of 
acceptability  of  a  materiel  item  for  Army  use  and  provides  a 
guide  to  authorization,  procurement,  logistical  support,  and 
asset  and  readiness  reporting.  See  AR  70-1,  Chapter  3,  for  type 
classification  designations  and  applicability. 

1-20.  DT&E  Requirements 

Type  classification  is  an  integral  part  of  the  MS  III  decision 
process.  Testing  and  program  documentation  requirements  are  the 
same  for  the  production  decision  and  the  type  classification 
designation.  As  a  minimum,  PPQT  must  be  completed  before  the 
full-rate  production  decision.  See  Part  Five  for  OT&E 
requirements . 


Section  VIII 

Joint  Acquisition  Program  DT&E 


1-21.  General 

DT&E  requirements  for  acquisition  programs  being  developed 
jointly  by  more  than  one  DOD  component  are  the  testing  procedures 
of  the  designated  lead  service.  Program  documents  are  developed 
by  the  lead  service  (see  AR  73-1,  Chapter  3) . 
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Section  IX 
Modifications 


1-22.  General 

Modifications,  whether  a  result  of  Preplanned  Product 
Improvement,  Engineering  Change  Proposals,  or  Post -Deployment 
Software  Support,  may  require  developmental  test  and 
evaluation/assessment.  The  purpose  of  testing  modifications  is 
to  determine  the  viability  and  adequacy  of  the  change  and  to 
determine  if  the  change  was  achieved  without  degradation  to  the 
system,  other  components,  and  interface  equipment.  Therefore, 
the  required  scope  and  type  of  testing/evaluation  varies  for  eac 
modification. 

1-23.  DT&E  Requirements 

To  determine  the  required  testing/ evaluation,  the  materiel 
developer,  in  coordination  with  the  combat  developer  and  the 
independent  evaluators/ assessors,  determines  which  of  the 
following  procedures  will  be  employed  for  the  conduct  of  T&E: 

a.  Through  the  TIWG  process  with  all  T&E  documented  in  the 
TEMP.  This  is  the  same  procedure  used  for  a  new  development 
program  and  includes  an  independent  developmental 
evaluation/assessment,  as  well  as  an  operational  evaluation. 

b.  Through  an  abbreviated  T&E  procedure  that  does  not  require 
a  TIWG,  a  TEMP,  or  an  operational  evaluation/ endorsement .  This 
procedure  may  or  may  not  require  developmental  testing  and  ^ 
evaluation.  This  is  determined  by  the  materiel  developer,  in 
coordination  with  the  TIWG.  The  materiel  developer  consults  the 
coordination  checksheet  (see  Part  One)  to  determine  if  ^ 
coordination  with  the  developmental  evaluator/assessor  is 
required.  If  coordination  is  required,  a  modification  package  is 
provided  to  the  evaluator/ assessor  for  determination  of  the  need 
to  conduct  a  formal  evaluation/assessment  and/or  requirement  to 
conduct  developmental  testing.  Waiver  may  occur  if  the  ^ 
checksheet  indicates  the  coordination  is  not  required  or  if  the 
developmental  evaluator/ assessor  (after  review  of  the 
modification  data)  decides  the  evaluation/assessment  is  not 
needed.  In  the  latter  case,  formal  notification  to  the  materiel 
developer  will  be  made  by  the  evaluator/  assessor.  In  both 
cases,  the  materiel  developer  will  furnish  the  independent 
evaluator/assessor  a  copy  of  the  completed  checksheet  for 
information. 

c.  A  major  modification  (i.e.,  a  program  that  meets  the 
criteria  of  an  ACAT  I  or  II  or  is  so  designated  by  the  decision 
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Section  X 

Waivers  of  Approved  Testing. 


1-24.  General 

DT&E  which  is  specified  in  the  approved  TEMP  must  be  conducted 
unless  a  waiver  has  been  obtained  from  the  TEMP  approval 
authority.  Procedures  for  requesting  waivers  can  be  found  in  AR 
73-1,  Chapter  3. 
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Chapter  2 

Requirements  Translation 

Section  I 
Introduction 

2-1.  General 

The  proper  interpretation  of  user  requirements  and  the  subsequent 
translation  of  those  requirements  to  testing  issues  and 
parameters  is  the  first  step  of  a  test  program.  Broad 
operational  capability  needs  are  translated  into  system— specif ic 
performance  requirements. 

Section  II 

Requirements  Documents 

2-2.  Mission  Need  Statement  (MNS) 

The  MNS  initiates  the  MS  0  review.  It  is  a  statement  of 
operational  capability  need  that  is  nonsystem  specific  and  states 
the  requirements  in  broad  operational  terms.  The  MNS  forms  the 
basis  for  the  Operational  Requirements  Document. 

2-3.  Operational  Requirement  Document  (ORD) 

The  ORD  contains  objectives  and  minimum  acceptable  requirements 
for  performance  tailored  to  each  concept.  They  are  generated  by 
the  combat  developer  and  coordinated  with  the  materiel  developer 
and  the  TIWG  members. 

Section  III 

Contractor  Specifications 

2-4.  Development  of  Contractual  Documents 

The  materiel  developer  generates  the  contractual  documents.  The 
translation  of  requirements  to  specifications  and  then  to  testing 
is  one  of  the  most  difficult  transitions  in  the  materiel 
acquisition  process.  Because  these  contractual  documents  must  be 
legally  exacting  and  enforceable  as  well  as  technically  complete, 
they  are  usually  more  voluminous  and  quite  different  than  the 
corresponding  requirements  document.  The  testers  and  evaluators 
aj-g  involved  in  the  development  of  these  documents  (i.e.  ,  the  RFP 
and  related  contractual  documents  such  as  the  system  and 
development  specifications)  through  the  review  process.  The  TIWG 
must  review  them  to  ensure  the  specifications  and  standards  cited 
are  correct,  the  proper  criteria  are  reflected,  and  the 
requirements  are  testable. 

2-5.  Role  of  Developmental  Tester  &  Evaluator/Assessor 
The  transition  from  requirements  document  to  contractual 
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requirements  is  a  key  in  developing  the  testing  program.  In 
particular,  the  government  developmental  tester  and 
evaluator/assessor  must  know  what  requirements  are  contained  in 
both  the  ORD  and  the  contract  to  formulate  a  testing  strategy 
that  takes  into  consideration  the  technical  requirements  and  the 
testing  performed  prior  to  acceptance  by  the  government.  This 
review  provides  the  TIWG  the  opportunity  to  take  maximum 
advantage  of  testing  resources.  This  review  also  facilitates  the 
test  data  confirmation  procedures  outlined  in  Section  ???, 

Chapter  4 . 

2-6.  Confirmation  of  the  Translation  Process 

When  the  contractor  receives  the  contractual  document  containing 
these  requirements,  there  is  another  translation  process.  This 
is  the  actual  fabrication  of  an  end  product  intended  to  meet  not 
only  the  technically  exacting  specifications  of  the  contract,  but 
also  the  program  baseline  requirements.  Government  developmental 
testing  provides  the  materiel  developer,  the  developmental 
evaluator/assessor ,  and  the  decision  maker  with  information  on 
the  contractor's  success  at  meeting  the  performance  standards  and 
establishes  the  safety  parameters  for  operational  testing.  In  a 
technical  sense,  it  is  a  feedback  loop  that  measures  what  was 
produced  by  the  contractor  against  what  was  intended  by  the 
contract.  It  is  important  because  it  allows  the  materiel 
developer  to  iterate  and  refine  the  product  when  problems  are 
revealed.  It  also  confirms  that  the  product  produced  is 
acceptable. 

Section  IV 

Critical  System  Characteristics 
2-7.  General 

The  ORD  identifies  the  Critical  System  Characteristics  with 
proposed  thresholds  and  objectives.  Critical  System 
Characteristics  are  those  design  features  that  determine  how  well 
the  proposed  concept  will  function  in  its  intended  operational 
environment.  They  include  survivability;  transportability; 
electronic  counter-countermeasures;  energy  efficiency;  and 
interoperability,  standardization,  and  compatibility  (DODI 
5000.2,  Part  4,  Section  C) .  Selected  Critical  System 
Characteristics  are  included  in  the  TEMP  as  Critical  Technical 
Parameters. 

Section  V 

Critical  Technical  Parameters 
2-8.  General 

Critical  technical  parameters  (CTP)  are  developed  jointly  by  the 
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independent  evaluator/assessor ,  the  materiel  developer,  and  the 
combat  developer,  with  input  from  the  TIWG 

The  CTPs  are  listed  in  matrix  format  in  Part  I  of  the  TEMP  (DOD 
5000. 2-M,  Part  7,  Figure  1). 

2-9.  Function  of  CTPs 

Each  CTP  has  measurable  objectives  and  thresholds  to  be 
evaluated,  and  they  are  derived  from  the  ORD,  the  critical  system 
characteristics  (including  software  maturity  and  performance 
measures),  and. the  technical  performance  measures.  CTPs 
establish  a  relationship  between  the  operational  requirements  and 
the  developmental  testing  to  be  performed  during  each  acquisition 
phase.  CTPs  are  evaluated  using  data  obtained  through  testing, 
surveys,  studies,  modeling  and  simulation,  or  other  analytica 
means . 


a.  Part  III  of  the  TEMP  includes  the  specific  critical 
technical  parameters  which  the  milestone  decision  authority  has 
designated  as  exit  criteria  and  which  must  be  confirmed  in  each 
phase  of  testing.  The  CTPs  must  take  into  consideration  the  COIC 
to  ensure  a  smooth  transition  from  DT&E  to  lOTE. 

b.  The  following  areas  will  be  considered  critical,  as 

appropriate:  system  performance,  physical  attributes, 

reliability,  availability,  and  maintainability;  system  safety; 
transportability;  health  hazards;  natural  environmental  or 
climatic  effects;  logistic  supportability ;  software; 
compatibility  and  interoperability;  survivability,  including 
conventional  ballistic  vulnerability;  nuclear  hardness  and 
survivability;  electromagnetic  environmental  effects;  directed 
energy  vulnerability;  chemical,  biological,  radiological 
vulnerability;  electronic  warfare,  countermeasures  and  counter¬ 
countermeasures;  training;  and  vulnerability  and  lethality. 

2-10.  Noncritical  technical  parameters 

Noncritical  technical  parameters  may  be  required  by  the 
developmental  evaluator/assessor  for  completeness  or  by 
regulatory  guidance  (such  as  generic  military  specifications) . 
Noncritical  parameters  may  become  critical  as  the  system  evolves. 
Noncritical  technical  parameters  are  documented  in  the  lEP/IAPs. 
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Chapter  3 

Developmental  Testing  and  Evaluation 


Section  I 

Role  of  the  Developmental  Tester 


3-1.  Contract  Requirements  _ 

Procurement  policies  define  testing  as  that  element  of  inspection 
which  determines  the  properties  or  elements,  including  functional 
operation  of  supplies  or  their  components,  by  the  application  or 
established  principles  and  procedures.  Inspection  is  defined  as 
"examining  and  testing  supplies  or  services  (including,  when 
appropriate,  raw  materials,  components,  and  intermediate 
assemblies)  to  determine  whether  they  conform  to  contract 
requirements . " 

3-2.  Review  of  Requirements 

The  contracting  officer  or  his/her  representative  has 
responsibility  for  all  contractual  matters.  However,  the 
materiel  developer  should  rely  on  the  developmental  tester  for 
assistance  in  ascertaining  that  the  technical  aspects  of  the 
system,  as  described  in  the  contractual  document,  are  properly 
tested.  For  this  reason,  the  materiel  developer  has  the 
government  developmental  tester  review  the  requirements  and 
testing  portions  of  both  the  solicitation  and  the  contractual 
documents.  Since  the  contract  designates  when  and  where  the 
Government  reserves  the  right  to  determine  that  the  technical 
requirements  in  the  contract  are  met,  the  government 
developmental  tester  provides  input  to  the  materiel  developer 
prior  to  issuance  of  the  solicitation  document.  This  review 
centers  on  the  requirements  relating  to  the  quality  of  the 
product  and  those  quality  controls  incumbent  on  the  contractor  to 
ensure  that  the  product  conforms  to  contractual  requirements. 

3-3.  Quality  Requirements  ...  ^  •  •4.-  i 

Because  of  the  complexity  of  Army  acquisitions  and  their  critical 
application  (in  which  failure  could  injure  personnel  or 
jeopardize  a  vital  mission) ,  quality  requirements  contained  in 
contractual  documentation  should  be  tested  by  the  Government 
developmental  tester. 

3-4.  Source  Selection  Evaluation  Board  (SSEB) 

The  Government  developmental  tester  should  a  be  an  advisor  to  and 
may,  if  appropriate,  participate  in  SSEB  deliberations.  In  this 
way,  testing  programs  can  be  structured  to  optimize  the 
acquisition  approach. 
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3-5.  Cost  Estimates 

When  a  government  contract  which  includes  testing  at  a  Government 
facility  already  exists,  the  request  for  cost  estimate,  the  test 
request,  and  the  test  funding  must  be  provided  through  the 
contracting  government  agency  directly  to  the  Government  tester 
rather  than  by  the  contractor.  If  a  prospective  contractor  is 
preparing  to  bid  on  a  government  contract  which  requires  testing, 
a  cost  estimate  for  testing  by  the  government  provided  to  that 
contractor  is  inappropriate.  This  would  constitute  privileged 
information  pertaining  to  the  solicitation.  Such  estimates  are 
obtained  and  distributed  to  all  bidders  by  the  contracting 
agency. 

3-6.  Private  Industry  Testing 

Test  services  may  be  provided  by  TECOM  for  private  industry  when 
no  related  acquisition  contract  exists.  The  guidance  and 
procedures  for  this  can  be  obtained  by  contacting  Commander, 
TECOM,  ATTN:  AMSTE-TA-0,  Aberdeen  Proving  Ground,  MD  21005- 
5055. 


Section  II 

Developmental  Evaluation/ Assessment 


3-7.  Continuous  Evaluation 

A  continuous  evaluation  process  is  used  on  all  acquisition 
programs  (see  Part  One) .  It  emphasizes  the  role  of  independent 
evaluation/assessments  throughout  the  acquisition  process  to 
ensure  responsible,  timely,  and  effective  evaluations/assessments 
of  the  status  of  a  system  as  it  progresses  into  mature  system 
effectiveness  and  suitability.  Early  involvement  of  the 
evaluators/assessors  in  the  acquisition  process  is  vital  to  a 
successful  developmental  program. 

3-8.  Developmental  Evaluation/Assessment  Responsibilities 
All  acquisition  categories  are  either  independently  evaluated  or 
independently  assessed  (see  AR  73-1,  paragraph  4-8).  At  program 
initiation  (MS  I) ,  a  decision  is  made  as  to  whether  the  program 
will  be  evaluated  or  assessed  based  on  the  following  criteria: 

a.  AMSAA  will  perform  developmental  independent  evaluations 
of  ACAT  I,  ACAT  II,  and  other  programs  as  designated  by  ASA(RDA) , 
and  associated  product  improvements,  TMDE,  ammunition,  system 
specific  support  items,  and  training  devices. 

b.  TECOM  will  perform  independent  developmental  assessments 
of  other  systems  (ACATs  III  and  IV)  and  associated  product 
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improvements,  TMDE,  ammunition,  system  specific  support  items, 
and  training  devices. 

c.  Once  assigned  to  either  AMSAA  or  TECOM,  the  responsibility 
for  assessment  or  evaluation  remains  with  that  organization 
throughout  the  life  cycle,  unless  reassigned  by  HQ  AMC  or  HQDA. 

d.  For  ACAT  III  and  IV  programs,  when  the  technology  or 
mission  of  the  system  is  so  similar  to  that  of  an  ACAT  I,  ACAT 
II,  or  another  ASA(RDA)-  designated  system  as  to  warrant 
evaluation,  it  may  be  determined  that  the  system  will  be 
evaluated  by  AMSAA. 

e.  For  those  ACAT  III  and  IV  systems,  when  the  level  of 
technology  or  the  requirements  for  modeling  and  simulation 
indicate  that  an  evaluation  would  be  appropriate,  the  decision 
may  be  made  that  an  evaluation  will  be  performed  by  AMSAA. 

3-9.  Basis  for  Evaluation/Assessment 

The  developmental  independent  evaluation/assessment  is  based  on 
test  data,  reports,  studies,  and  other  appropriate  sources,  and 
is  the  culmination  of  a  major  effort  by  many  people.  The 
evaluat ion/assessment  compares  the  performance  of  the  item,  as 
derived  from  the  test  data,  directly  with  the  requirements 
reflected  in  the  ORD  and  the  Critical  Technical  Parameters 
contained  in  the  TEMP.  The  independent  evaluator/assessor  also 
compares  the  test  results  to  the  required  operational  performance 
according  to  the  mission  profile,  the  Cost  &  Operational 
Effectiveness  Analysis  (COEA) ,  and  to  any  past  results,  and 
addresses  the  "so  what”  questions  concerning  requirements. 

3-10.  Procedures/Techniques  for  Evaluation/Assessment 
Through  the  lEP/IAP  or  the  TDP,  the  evaluators/assessors  identify 
the  procedures  and  techniques  which  will  be  used  in  the 
evaluation.  For  example, 

a.  Specific  models  will  -be  identified,  e.g.,  reliability 
growth,  performance,  logistics.  (See  Chapter  16,  Part  One.) 

b.  Method  of  statistical  analysis  (inferential  techniques) 
are  identified,  such  as  hypothesis  testing  and  analysis  of 
variance. 

c.  Comparison  of  results  to  the  requirements  may  be 
accomplished  through  quantitative  comparison  or 

sub j ecti ve/qualitative  comparison . 

d.  Measures  of  effectiveness  are  developed,  e.g.,  system 
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effectiveness  (Ego)  ,  fractional  target  damage,  probability  of  a 
kill  given  a  hit,  probability  of  a  hit,  probability  of  detection. 


Section  III 

Role  of  the  Developmental  Independent  Evaluator/Assessor 


3-11.  General 

The  evaluators/assessors  contribute  to  the  acquisition  and 
fielding  of  an  effective,  supportable,  and  safe  system  by 
assisting  in  the  engineering  design  and  development  and  verifying 
attainment  of  technical  performance  specifications,  objectives, 
producibility,  adequacy  of  the  Data  Package,  supportability ,  RAM 
and  MANPRINT  aspects.  Developmental  evaluation  encompasses  data 
obtained  from  the  use  of  models,  simulations,  and  test  beds,  as 
well  as  prototypes  or  full-scale  development  models  of  the 
system . 

3-12.  Reporting  Evaluations/Assessments 

For  each  milestone  decision  review  and  the  materiel  release 
decision,  the  developmental  evaluator /assessor  prepares  an 
lER/IAR.  Coordination  of  information  contained  in  the  lER/IAR 
between  the  testers  and  evaluators  is  important  to  ensure  that 
test  data  are  accurately  reflected.  The  IER\IAR  highlights  those 
technical  parameters  that  have  been  evaluated  and  require  no 
additional  analysis.  They  also  present  those  that  need  further 
evaluation  to  support  MS  decisions,  as  well  as  materiel  release 
actions. 

3-13.  Mission  of  Evaluators/Assessors 

The  developmental  independent  evaluators/assessors  serve  on  the 
following  working  groups,  committees,  or  teams,  if  formed: 

Special  Task  Force,  Special  Study  Groups  (AR  71-9) ,  TIWGs,  TRRs, 
RAM  scoring  conferences  and  RAM  assessment  conferences,  MANPRINT 
Joint  Working  Group,  System  Safety  Working  Group,  market  survey 
teams,  and  ILSMTs.  Developmental  independent 

evaluators/assessors  normally  brief  the  lER/IAR  at  pre-ASARCs  and 
IPRs  for  materiel  systems  and  at  pre-MAISRCs  for  information 
systems.  Developmental  lER/IARs  as  well  as  the  operational  lER 
are  briefed  by  the  operational  IE  at  the  ASARCs  and  the  MAISRCs. 
Specific  actions  to  be  performed  or  participated  in  by  the 
developmental  IE  are  — 

a.  Review  and  be  knowledgeable  of  applicable  MAA  results. 

b.  Review  requirements  documents.  System  Program  Issues, 
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critical  issues,  and  the  System  Threat  Assessment  Report  (STAR) . 

c.  Provide  information  to,  and  draw  information  from,  the  TIR 
data  base  (see  Capstone  Volume) . 

d.  Review  system  support  packages. 

e.  Develop  technical  parameters  and  process  for  approval  as 
appropriate. 

'f.  Participate  in  the  development  of  system  acquisition 
strategies,  and  assist  in  the  development  of  test  portions  of 
selected  Requests  for  Proposal  (RFP) . 

g.  Within  the  TIWG,  develop  procedures  to  accept  data 
gathered  from  all  sources. 

h.  Perform  developmental  test  integration  as  a  member  of  the 
TIWG,  assist  in  the  development  of,  and  provide  concurrence  in, 
the  TEMP,  and  chair  the  LFTE  Working  Group,  if  required. 

i.  Review  and  comment  on  test  plans. 

j .  Respond  to  questions  and  provide  input  in  terms  of 

(  performance  analysis  during  the  development  of  the  COEA,  cost  and 

training  effectiveness  analysis  (CTEA) ,  and  other  cost  analyses. 

k.  Review  and  comment  on  the  System  MANPRINT  Management  Plan 
and  the  ILS  Plan. 

l.  Develop  lEP/IAPs,  Test  Design  Plans  (TDPs) ,  and  lER/IARs. 

m.  Review  and  provide  concurrence  in  the  RAM  Rationale 
Report . 

n.  Determine  and  coordinate  modeling  and  simulation  needs  and 
use  their  results. 

o.  Monitor  tests  in  coordination  with  the  developmental 
tester. 

p.  Recommend  approval/disapproval  of  requests  for  waiver  of 
approved  tests  and  recommend  test  suspensions  when  required. 

q.  Evaluate  Test  Incident  Reports  (TIR)  and  corresponding 
corrective  actions. 

r.  Advise  in  source  selection  activities,  as  required  by  the 
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materiel  developer. 

s.  Present  results  of  Developmental  lER/IAR  at  Operational 
Test  Readiness  Reviews  (OTRR) ,  as  appropriate. 

The  developmental  lE/IAs  continue  involvement  throughout  the 
system  life  cycle  by  monitoring  key  deployment  activities.  This 
is  accomplished  by  partici-  pation  in  system  performance  reviews; 
fielded  equipment  reviews  (e.g.,  Fielded  System  Review); 
research,  development,  and  acquisition  (RDA)  reviews;  and  sample 
data  collection.  It  is  also  accomplished  by  monitoring  stockpile 
reliability  and  lot  acceptance  tests. 


Section  IV 

Role  of  System  Contractor  in  Support  of _Developmental  Testing 


3-14.  General 

The  objectives  of  developmental  testing  and  evaluation  include 
verifying  system  maturity,  logistic  supportability ,  human 
factors,  and  system  safety.  One  of  the  primary  objectives  is  to 
verify  reliability,  availability,  maintainability-durability 
(RAM-D)  maturity.  Therefore,  testing  is  designed  to  find, 
analyze,  and  fix  problems  and  verify  the  solutions.  Meeting 
these  objectives  requires  engineering  level  involvement  of,  and 
discussions  with,  system  contractor  personnel. 

3-15.  Nature  of  Contractor  Involvement 

The  degree  and  nature  of  system  contractor  involvement  in  the 
developmental  testing  must  be  agreed  to  by  the  developer,  the 
evaluator,  the  government  tester,  and  all  others  concerned, 
preferably  through  the  TIWG.  Those  agreements  must  then  be 
communicated  through  contractual  requirements.  Developing  these 
agreements  early  will  help  to  ensure  that  test  data  will  be 
usable  for  evaluation  purposes.  (See  Section  XII,  Test  Data 
Confirmation. ) 

a.  Contractor  involvement  may  range  from  no  direct 
involvement  to  providing  spare  parts  and  technical  advice  during 
the  conduct  of  the  test,  to  performing  entire  tests.  When  the 
contractor  will  be  directly  involved  in  the  conduct  of  testing  at 
a  government  facility,  special  consideration  may  be  required  to 
address  security,  personnel  safety,  and  the  protection  of 
competition  sensitive  test  data.  When  the  contractor  will 
perform  the  testing,  consideration  should  be  given  to  the  use  of 
a  combined  government/ contractor  developmental  test  team.  Use  of 
the  team  provides  for  government  participation  in  the  development 
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of  the  contractor  test  plans.  The  test  results  are  reported  by 
the  contractor  and  verified  by  the  government  test  personnel, 
thus  avoiding  duplication  of  testing. 

b.  The  degree  of  contractor  involvement  in  scoring/assessment 
conferences  dealing  exclusively  with  developmental  test  and 
evaluation  will,  likewise,  be  determined  by  the  materiel 
developer  in  coordination  with  the  TIWG.  Contractor  personnel, 
in  general,  should  not  be  physically  present  during  the  formal 
voting/ scoring/  assessment  period.  However,  the  presence  of 
contractor  personnel  may  be  allowed  during  formal  scoring  at 
developmental  scoring  conferences  if  it  is  considered  necessary 
for  proper  information  flow,  except  as  discussed  in  the  following 
paragraph . 

c.  In  those  cases  where  it  is  known  that  the  developmental 
test  will  be  conducted  under  conditions  similar  (operational  mode 
summary /mission  profile,  stresses,  environmental  conditions,  test 
support,  fixed  and  same  configuration)  to  lOT&E,  and  lOT&E  is  to 
be  conducted  during  the  same  phase,  OPTEC  will  notify  AMC  that 
the  developmental  test  results  are  to  be  combined  with  the  lOT&E 
results.  If  agreed  to  by  the  materiel  developer,  system 
contractor  participation  in  the  scoring  and  RAM  assessment 
conferences  addressing  developmental  test  data  will  be  the  same 
as  for  lOT&E  conferences.  (Reference  Part  One,  Chapter  11.) 


Section  V 

Developmental  Test  Types 


3-16.  General 

Developmental  tests  are  categorized  as  reflected  in  AR  73-1, 
Chapter  4.  A  definition  and  brief  description  of  the 
developmental  tests  performed  throughout  the  acquisition  cycle 
follows.  The  tests  are  separated  into  the  pre-MS  III, 
production,  and  post-production  phases.  The  software  tests 
defined  here  are  Software  Qualification  Test  (SQT)  and  Post 
Deployment  Software  Support  (PDSS) .  For  other  developmental 
tests  conducted  by  the  developer  of  the  software,  see  Part  Seven. 

3-17.  Pre-MS  III  Developmental  Test  Types 

Pre-MS  III  developmental  testing  ranges  from  program  initiation 
to  the  MS  III  production  decision  and  includes  funding  categories 
6. 1-6. 4. 


a.  Research  Effort/Test.  Developmental  effort/test  conducted 
prior  to  MS  o  to  determine  early  technical  parameters,  to  support 
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the  research  of  these  items,  and  to  provide  fundamental  knowledge 
for  solutions  of  identified  military  problems.  Program  funding 
category  -  6.1  and  6.2. 

b.  Technical  Feasibility  Test  (TFT).  DT  conducted  post-MS  0, 
pre-MS  I  or  MS  I/II  to  assist  in  determining  safety,  establishing 
system  performance  specifications,  and  determining  feasibility  of 
alternative  concepts.  Testing  during  the  Concept  Exploration  and 
Definition  Phase  identifies  and  reduces  risks  in  subsequent 
acquisition  phases.  This  test  provides  data  for  the  independent 
developmental  evaluation/assessment  which  supports  the  MS  I  or  MS 
I/II  decision.  Program  funding  category  -  6.3.  NOTE:  While  not 
tied  to  specific  acquisition  programs,  the  following  Technology 
Base  demonstrations  may  be  conducted  by  the  Government 
developmental  tester.  Technology  demonstrations  are  conducted  to 
assess  the  military  utility  or  cost  reduction  potential  of 
innovative  Government  or  commercially  developed  technology  (DODI 
5000.2,  Part  5-C) . 

(1)  Advanced  Technology  Transition  Demonstration  (ATTD) . 
ATTDs  are  used  to  expedite  technology  transition  from  the 
laboratory  to  operational  use.  They  are  demonstrations  conducted 
in  an  operational  environment  and  are  primarily  funded  with  6.3A 
funds.  These  demonstrations  may  integrate  advanced  technologies 
to  establish  the  feasibility  of  a  concept  or  may  utilize 
prototypes,  surrogates,  and  simulations  to  show  that  existing 
technology  can  support  a  concept.  ATTDs  should  include 
provisions  for  early  testability  and  operational  assessments. 

(2)  Proof  of  Principle  (POP)  demonstrates,  in  a 
nonoperational  environment,  innovative  technologies  that  will 
support  system  upgrades  or  provide  new  operational  capabilities. 
POPs  are  technical  demonstrations  and  troop  experimentations 
conducted  with  brassboard  configurations,  subsystems,  or 
surrogate  systems. 

c.  Engineering  Development  Test  (EDT) .  DT  conducted  post-MS 
I,  pre-MS  II  to  demonstrate  attainability  of  critical 
technologies  and  processes  and  to  define  design  characteristics 
and  capabilities,  including  ruggedness  of  hardware 
configurations,  compatibility  and  interoperability  with  existing 
or  planned  systems,  and  the  effects  of  natural  and  induced 
environmental  conditions.  This. test  provides  data  for  the 
independent  developmental  evaluation/  assessment  which  supports 
the  MS  II  decision.  Program  funding  category  -  6.3. 

d.  Production  Proveout  Test  (PPT) .  A  DT  conducted  with 
prototype  hardware  post-MS  II  or  post-MS  I/II,  prior  to  MS  III  to 
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select  the  most  appropriate  source  or  design.  When  MS  I  and  MS 
II  are  combined,  PPT  may  also  be  used  to  provide  data  on  safety, 
the  achievability  of  critical  system  technical  parameters, 
refinement  and  ruggedization  of  hardware  and  software 
configurations,  and  determination  of  technical  risks.  Program 
funding  category  -  6.4. 

e.  Preproduction  Qualification  Test  (PPQT) .  A  system  level 
DT  conducted  post-MS  II  or  post-MS  I/II  (usually  just  prior  to  MS 
III)  that  demonstrates  design  integrity  over  the  specified 
operational  and  environmental  range.  These  tests  usually  use 
prototype  or  preproduction  hardware  and  software  fabricated  to 
the  proposed  production  design  specifications  and  drawings.  Such 
tests  include  contractual  reliability  and  maintainability 
demonstration  tests  required  prior  to  production  release.  This 
test  provides  data  for  the  independent  developmental 
evaluation/assessment  which  supports  the  MS  III  production 
decision.  Program  funding  category  -  6.4. 

(1)  The  objectives  of  the  PPQT  are  to  have  the  government 
confirm  the  design  is  stable,  logistically  supportable,  capable 
of  being  produced  efficiently  and  will  meet  the  performance/user 
requirements,  assess  the  performance  envelope,  and  determine  the 
adequacy  of  any  corrective  action  indicated  by  previous  tests. 

(2)  PPQT  may  also  include  tests  which  are  not  included  in 
the  data  package  or  contract  (example:  environmental  extremes, 
test-to-failure)  when  necessary  to  obtain  engineering  data  for 
corrective  action  verification  or  other  purposes.  PPQT  may  be 
accomplished  in  phases  (e.g.,  preliminary  engineering,  specific 
problem  correction,  etc.).  • 

(3)  For  those  weapons  systems  required  by  law  to  undergo 
live  fire  test  and  evaluation  (see  volume  5) ,  the  live  fire  test 
(LFT)  is  conducted  as  part  of  or  in  conjunction  with  the  PPQT. 

The  LFT  demonstrates  the  ability  of  the  system  to  provide  battle 
resilient  survivability,  or  the  munition  to  provide  lethality. 

It  will  provide  insights  into  the  principal  damage  mechanisms  and 
failure  modes  occurring  as  a  result  of  the  munition/target 
interaction  and  into  techniques  for  reducing  personnel  casualties 
or  enhancing  system  survivability/ lethality.  LFT  is  conducted 
with  6.4  or  procurement  funding. 

f.  Logistic  Demonstration  (LD) .  An  LD  is  a  nondestructive 
disassembly  and  reassembly  of  equipment  conducted  on  a  dedicated 
engineering  prototype  prior  to  MS  III  to  evaluate  the 
supportability  of  the  materiel  design  and  the  preliminary  system 
support  package.  It  evaluates  the  achievement  of  maintainability 
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goals;  the  adequacy  and  sustainability  of  tool,  test  equipment, 
technical  publications,  maintenance  instructions,  and  personnel 
skill  requirements;  the  selection  and  allocation  of  spare  parts, 
tools,  test  equipment,  and  tasks  to  appropriate  maintenance 
levels;  and  the  adequacy  of  maintenance  time  standards.  Program 
funding  category  -  6.4. 

g.  Software  Qualification  Test  (SQT) .  The  SQT  is  a  system 
level  test  conducted  by  the  Army  developmental  tester  using  live 
data  files  supplemented  with  user-prepared  data  and  executed  on 
target  hardware.  Conversion  procedures  and  special  training 
requirements  are  introduced  as  additional  elements  for 
verification  and  validation.  The  objectives  of  the  SQT  are  to 
have  the  government  confirm  the  design  will  meet  the 
performance/user  requirements  and  to  determine  the  adequacy  of 
any  corrective  action  indicated  by  previous  testing.  System 
users  participate  in  the  technical  and  functional  aspects  of  the 
SQT. 

3-18.  Production  testing 

Production  testing  is  required  to  determine  the  producer's 
performance  in  producing  items  that  meet  prescribed  TDP  require¬ 
ments,  to  provide  test  data  for  materiel  release  action,  and  to 
ensure  that  the  products  continue  to  meet  the  prescribed 
requirements.  Program  funding  category  -  Procurement. 

a.  First  Article  Test  (FAT) .  FATs  are  conducted  post-MS  III 
to  ensure  the  contractor  can  furnish  a  product  that  conforms  to 
all  contract  require-  ments  for  acceptance  (Federal  Acquisition 
Regulation  (FAR),  Subpart  9.3).  Requirements  for  FATs  are 
invoked  in  production  contracts  by  citation  of  the  applicable  FAR 
first  aritcle  inspection  and  approval  clause.  When' a  FAT  is 
specified  in  a  contract,  it  may  not  be  waived  or  changed  without 
prior  approval  of  the  head  of  the  contracting  activity.  Tests 
may  be  conducted  at  government  facilities  or  at  contractor 
facilities  when  observed  by  the  govern-ment.  Requirements  for 
the  FAT  should  be  consistent  with  those  of  the  PQT. 

b.  Production  Qualification  Test  (PQT) .  A  system  level  DT 
conducted  post-MS  III  to  verify  that  the  production  item  meets 
contract  requirements,  determine  the  adequacy  of  any  corrective 
action  indicated  by  previous  (pre-MS  III)  tests,  and  validate  the 
manufacturer's  facilities,  procedures,  and  processes.  Within  the 
Army,  this  test  provides  data  for  the  independent  evaluation  for 
materiel  release  so  that  the  evaluator  can  address  the  adequacy 
of  the  materiel  with  respect  to  the  stated  requirements. 

Materiel  release  is  accomplished  during  the  first  post-MS  III 
production  contract  and  is  repeated  if  the  process  or  design  is 
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significantly  changed,  if  a  second  source  for  the  system  or  major 
components  therein  is  brought  on  line,  or  if  a  significant  break 
in  production  occurs.  PQT  will  also  provide  a  baseline  for  post¬ 
production  testing.  The  PQT  may  include  tests  which  are  not 
included  in  the  data  package  or  contract  (e.g. ,  environmental 
extremes,  test-to-failure)  when  necessary  to  obtain  engineering 
data  for  corrective  action  verification. 

c.  Follow-on  Production  Test  (FPT) .  A  DT  conducted  on  full 
production  models  to  verify  the  adequacy  of  corrective  actions 
indicated  by  the  PQT  or  to  determine  production  acceptability. 
FPTs  are  structured  similarly  to  PQTs. 

d.  Comparison  Test  (CPT) .  A  CPT  is  a  test  of  random  samples 
from  production  and  is  conducted  as  a  quality  assurance  measure 
to  detect  any  manufacturing  or  quality  deficiencies  that  may  have 
developed  during  volume  production  which  could  reduce  effective 
operation  of  the  item  or  result  in  item  degradation.  CPT  is 
conducted  or  supervised  by  an  agent  independent  of  the  producer 
or  Government  on-site  quality  assurance  personnel.  CPT  may  be 
conducted  at  procuring  agency  facilities.  Government  testing 
installations,  or  contractor  facilities. 

e.  Quality  Conformance  (Acceptance)  Inspections.  These 
inspections  are  examinations  and  verification  tests  normally 
prescribed  in  the  TDP  for  performance  by  the  contractor  and 
subject  to  performance  or  witnessing  by  the  on-site  Quality 
Assurance  Representative  on  the  items,  lots  of  items,  or  services 
to  be  offered  for  acceptance  under  a  contract  or  purchase  order. 
These  examinations  and  tests  include,  as  necessary,  in-process 
and  final  measurements  or  comparisons  with  technical  quality 
characteristics  required  to  verify  that  materiel  meets  all  the 
terms  of  the  contract  and  should  be  accepted  by  the  Government. 

f.  PDSS  Test.  Developmental  test  that  is  conducted  during 
post-deployment  software  support  to  assure  that  software 
modifications  meet  requirements,  do  not  impair  existing  functions 
or  performance,  can  be  employed  by  users,  and  are  effective  and 
suitable. 

3-19.  Post-production  testing 

Post-production  DT  is  conducted  to  assure  that  materiel,  which  is 
stored,  reworked,  repaired,  renovated,  rebuilt,  or  overhauled 
after  initial  issue  and  deployment,  conforms  to  specified 
quality,  reliability,  safety,  and  operational  performance 
standards.  Program  funding  category  -  O&MA. 

a.  Surveillance  Tests.  Include  destructive  or  nondestructive 
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tests  of  materiel  in  the  field  or  in  storage  at 

field/depot/extreme  environmental  sites.  They  are  conducted  to 
determine  suitability  of  fielded  or  stored  materiel  for  use; 
evaluate  the  effects  of  environments;  measure  deterioration; 
identify  failure  modes;  and  to  establish/predict  service  and 
storage  life.  Surveillance  test  programs  may  be  at  the 
component-through-system  level.  System  level  programs  may 
include  dedicated  hardware  allocated  for  this  purpose,  fielded 
materiel,  or  supplies  in  storage.  Storage  sites  may  include 
depots,  field  storage,  or  extreme  environmental  locations. 
"Libraries”  of  component  parts  to  provide  a  baseline  for 
subsequent  surveillance  test  data  comparisons  may  be  established 
at  contractor  or  government  facilities. 

b.  Reconditioning  Tests.  The  reconditioning  test  type 
includes  the  following  categories: 

(1)  Pilot  reconditioning  tests  are  conducted  to 
demonstrate  the  adequacy  of  the  documented  technical 
requirements,  processes,  facilities,  equipment,  and  materials 
that  will  be  used  during  volume  reconditioning  activities.  The 
pilot  model  will  be  reconditioned  in  strict  accordance  with  the 
above.  Pilot  reconditioning  testing  relates  to  PQTs  during 
production.  Pilot  reconditioning  tests  will  be  applied  when 
DMWRs,  TMs,  or  TBs  are  used  the  first  time  or  when  major  changes 
are  made. 

(2)  Initial  reconditioning  tests  are  conducted  to 
demonstrate  the  quality  of  the  materiel  when  reconditioned  under 
volume  (rate)  procedures  and  practices.  These  tests  relate  to 
FATS  during  production.  Initial  recon-  ditioning  tests  will  be 
conducted  when  an  item  is  reconditioned  for  the  first  time  by  a 
government  or  contractor  facility,  when  changes  in  processes  or 
facilities  occur,  or  when  there  has  been  a  significant  break  in 
reconditioning  operations. 

(3)  Control  Tests.  Control  tests  are  conducted  on 
randomly  selected  items  from  volume  reconditioning  operations  to 
verify  that  the  process  is  still  producing  satisfactory  materiel. 
Criteria  should  be  the  same  as  for  initial  reconditioning  tests. 
These  tests  relate  to  CPTs  during  production. 

(4)  Acceptance  Tests.  Acceptance  tests  are  conducted  on 
in-process  materiel  and  at  the  completion  of  reconditioning 
activities,  and  upon  which  an  accept/reject  decision  is  based. 

(5)  Baseline  Evaluation  Test  (BET) .  BETs  are  conducted 
simul-  taneously  on  reconditioned  and  new  production  materiel  of 
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the  same  configuration  to  provide  a  comparison  of  performance  and 
to  determine  the  degree  of  reconditioning  required.  BET  will  be 
considered  when  the  item  is  being  reconditioned  for  the  first 
time,  when  significant  modifications  affecting  performance  are 
incorporated,  or  to  provide  data  upon  which  to  base  a  decision  on 
upgrading  versus  new  procurement. 


Section  VI 

D^velopmental  Test  Facilities 


3-20.  General 

The  Army  maintains  and  operates  a  group  of  test  centers  for  the 
efficient  accomplishment  of  developmental  testing  for  research 
and  throughout  all  phases  of  acquisition.  These  test  centers 
have  evolved  as  specialized  and  general  purpose  ranges  and  test 
facilities,  with  capabilities  which  cover  the  full  range  of  Army 
systems.  The  capabilities  of  each  of  the  Army  developmental  test 
facilities  is  described  briefly  in  the  following  paragraphs.  The 
descriptions  are  not  meant  to  be  all-inclusive.  Additional 
detail  may  be  obtained  directly  from  the  test  center's  parent 
command . 

3-21.  Combat  Systems  Test  Activity 

U.S.  Army  Combat  Systems  Test  Activity  (USACSTA) ,  Aberdeen 
Proving  Ground,  Maryland,  is  a  Major  Range  and  Test  Facility  Base 
activity  of  the  U.S.  Army  Test  and  Evaluation  Command.  USACSTA 
tests  vehicles,  armors,  munitions,  weapons,  general  equipment, 
robotic  vehicles,  and  individual  clothing  and  equipment. 
Facilities  include  instrumented  firing  ranges  for  small  arms, 
mortars,  artillery  out  to  22,000  meters,  and  tank  gunnery;  all 
standardized  vehicle  test  courses;  live  fire  test  facilities  for 
vehicles,  ship  structures,  and  explosive  targets  involving 
depleted  uranium  armor  or  projectiles;  and  a  full  array  of 
environmental  facilities  to  satisfy  the  testing  requirements  of 
MIL-STD  810  as  well  as  electromagnetic  interference,  and  pulse 
radiation.  Unique  facilities  include  moving  target  simulators, 
high  energy  and  flash  radiography,  and  waterway  related  test 
ranges. 

3-22.  Dugway  Proving  Ground 

U.S.  Army  Dugway  Proving  Ground  (DPG) ,  Dugway,  Utah,  is  a  Major 
Range  and  Test  Facility  Base  activity  of  the  U.S.  Army  Test  and 
Evaluation  Command.  DPG  tests  chemical  and  biological  materiel, 
smoke,  obscurants,  and  incendiary  devices,  artillery  and  mortars, 
and  tropic  natural  environmental  effects  on  all  materiel. 
Facilities  include  instrumented  outdoor  test  grids  to  measure 
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effectiveness  of  smokes,  obscurants,  and  dispersal  of  chemical 
munitions  using  simulants;  chemical  and  biological  laboratories; 
an  Indoor  test  chamber  to  subject  systems  as  large  as  a  tank  to 
chemical,  biological,  and  environmental  challenges;  and  mortar 
and  artillery  ranges  out  to  65,000  meters.  Facilities  also 
include  a  remote  test  site  in  Panama  to  test  the  full  range  of 
Army  weapons  systems,  clothing  and  individual  equipment  for 
effects  of  operation  and  long  term  exposure  in  natural  tropical 
environments . 

3-23.  Electronic  Proving  Ground 

U.S.  Army  Electronic  Proving  Ground  (USAEPG) ,  Fort  Huachuca, 
Arizona,  is  a  specialized  Major  Range  and  Test  Facility  Base 
activity  of  the  U.S.  Army  Test  and  Evaluation  Command.  USAEPG 
tests  systems  with  regard  to  communications,  command  and  control, 
optics  and  electro-optics,  intelligence,  electronic  warfare, 
avionics,  and  TEMPEST.  Facilities  include  an  instrumented  test 
range,  an  electromagnetic  environmental  test  facility, 
environmental  facilities  to  satisfy  the  requirements  of  MIL-STD 
810,  a  stress  loading  facility  to  provide  a  threat 
electromagnetic  environment  and  measure  the  full  load  performance 
of  communications  systems,  and  many  unique  specialized  facilities 
for  testing  of  antennas,  radars,  remotely  piloted  vehicles,  and 
computer  software. 

3-24.  White  Sands  Missile  Range 

U.S.  Army  White  Sands  Missile  Range  (WSMR) ,  New  Mexico,  is  a 
Major  Range  and  Test  Facility  Base  activity  of  the  U.S.  Army  Test 
and  Evaluation  Command.  WSMR  tests  missile  systems  and  related 
materiel,  air  defense  systems,  laser  weapons  systems,  and  nuclear 
effects  on  all  systems.  Facilities  include  on-range  and  off- 
range  missile  launch  facilities  providing  up  to  800  miles  over¬ 
land  trajectory,  flight  ranges  highly  instrumented  with  radars, 
cinetheodolites,  telemetry,  optics,  laser  trackers  and  command 
control  and  command  destruct  systems,  a  laser  test  range,  and 
target  drone  control  facility.  Specialized  environmental 
facilities  provide  nuclear  effects,  electromagnetic  radiation, 
microbiological,  climatic  and  dynamic  test  environments. 

3-25.  Yuma  Proving  Ground 

U.S.  Army  Yuma  Proving  Ground  (YPG) ,  Yuma,  Arizona,  is  a  Major 
Range  and  Test  Facility  Base  activity  and  the  desert  natural 
environmental  test  center  of  the  U.S.  Army  Test  and  Evaluation 
Command.  YPG  tests  long  range  tube  artillery,  automotive 
systems,  tank  armament,  aircraft  armament,  air  delivery,  air 
transport,  remotely  piloted  vehicles,  and  natural  desert 
environmental  effects  on  all  weapons  systems  and  materiel. 
Facilities  include  fully  instrumented  land  and  water  air  delivery 
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drop  zones,  firing  ranges  from  small  arms  to  artillery  out  to 
75,000  meters,  air  to  ground  aircraft  armament  range,  tank 
gunnery  range,  navigation  system  range,  and  a  full  array  of 
ground  vehicle  mobility  test  courses.  The  hot-dry  natural  desert 
environment  provides  diverse  terrain  representative  of  almost  all 
of  the  world's  desert  areas. 

3-26.  Kwajalein  Atoll 

U.S.  Army  Kwajalein  Atoll  (USAKA) ,  Republic  of  the  Marshall 
Islands,  is  a  Major  Range  and  Test  Facility  Base  activity  of  the 
U.S.  Army  Strategic  Defense  Command.  USAKA  provides  range,  radar 
tracking,  impact  scoring,  recovery,  and  telemetry  data  collection 
for  intercontinental  ballistic  missiles,  orbital  objects,  and 
reentry  vehicles.  Facilities  include  a  broad  range  of 
instrumentation,  tracking,  imaging,  and  splash  detection  radars, 
hydroacoustic  impact  timing,  and  optical  and  video  sensors.  The 
natural  configuration  of  the  atoll  facilitates  tracking  and 
recovery  of  reentry  vehicles. 

3-27.  Cold  Regions  Test  Center 

U.S.  Army  Cold  Regions  Test  Center  (USACRTC) ,  Fort  Greely, 

Alaska,  is  the  natural  cold  environmental  test  center  of  the  U.S. 
Army  Test  and  Evaluation  Command.  CRTC  conducts  basic  cold 
environment  tests  on  all  materiel  as  prescribed  by  AR  70-38. 
Facilities  include  artillery  ranges  to  55,000  meters,  tank  ranges 
to  4000  meters,  vehicle  courses,  chemical  (simulant)  and  smoke 
test  grids,  mobile  instrumentation  vans,  ski  trails,  and  large 
expanses  in  which  to  test  full  systems  operationally  in  the 
natural  winter  environment.  Conditions  include  snow  to  seven 
feet  deep,  ice  fog,  permafrost  tundra,  temperatures  in  the  -5°  to 
-25°  F  range  during  most  of  the  winter,  with  temperatures  often 
dipping  below  -50°  F. 

3-28.  Jefferson  Proving  Ground 

U.S.  Army  Jefferson  Proving  Ground  (JPG),  Madison,  Indiana,  is  a 
test  center  of  the  U.S.  Army  Test  and  Evaluation  Command.  JPG 
performs  primarily  production  and  post-production  testing  of 
artillery  ammunition  and  ammunition  components,  bombs,  grenades, 
and  conventional  and  smart  mines.  Facilities  include  munitions 
assembly  plants  and  environmental  conditioning  facilities, 
instrumented  artillery  ranges  to  23,000  meters,  specialized 
ranges  and  firing  positions  for  direct  fire  testing,  weapons, 
smoke  and  illumination  projectiles,  fuzes,  depleted  uranium 
ammunition,  and  weapons  proofing. 

3-29.  Aviation  Technical  Test  Center 

U.S.  Army  Aviation  Technical  Test  Center  (USAATTC) ,  Fort  Rucker, 
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Alabama,  is  a  test  center  of  the  U.S.  Army  Test  and  Evaluation 
Command.  USAATTC  tests  performance,  suitability,  and 
airworthiness  of  fixed  and  rotary  wing  aircraft,  aircraft 
components,  subsystems,  and  related  ground  support  equipment. 
Facilities  include  a  static  engine  test  cell,  inflight 
performance  recorders,  and  ranges  for  the  firing  of  aircraft 
weapons  systems.  Facilities  also  include  an  airworthiness 
qualification  test  site  at  Edwards  Air  Force  Base  for  the  conduct 
of  safety  and  technology  test  and  evaluation  of  aircraft  systems 
and  subsystems. 

3-30.  Redstone  Technical  Test  Center 

U.S.  Army  Redstone  Technical  Test  Center  (USARTTC) ,  Redstone 
Arsenal,  Alabama,  is  a  test  center  of  the  U.S.  Army  Test  and 
Evaluation  Command.  USARTTC  tests  small  rockets  and  guided 
missiles.  Facilities  include  fully  instrumented  flight  ranges  up 
to  8000  meters,  1000-  and  2000-meter  dynamic  test  sled  tracks, 
static  rocket  motor  test  facilities  for  up  to  10  million  pounds 
of  thrust,  and  electromagnetic,  lightning,  and  portable  climatic 
environmental  facilities  for  testing  missiles,  rockets,  and 
explosive  warheads.  Facilities  are  also  available  for  testing  of 
electro-optical  devices  under  simulated  battlefield 
countermeasures  and  obscurants. 


Section  VII 

Requesting  Technical  Test  Services 


3-31.  General 

This  section  provides  procedures  for  requesting  developmental 
test  services  from  U.S.  Army  Test  &  Evaluation  Command  (TECOM) . 
It  includes  procedures  for  requesting  developmental  testing  and 
related  support  requirements  as  well  as  procedures  for 
identifying  future  requirements  involving  TECOM  facilities  and 
resources. 

3-32.  Program  Planning  Forecasts 

a.  The  Program  Planning  Forecast  is  a  mechanism  designed  to 
identify  future  testing  requirements.  It  serves  a  dual  purpose: 
providing  a  forecast  of  requirements  for  developmental  testing 
from  the  materiel  developer  to  TECOM  and  providing  a  preliminary 
budget  estimate  and  test  schedule  from  TECOM  to  the  materiel 
developer.  The  planning  forecast  is  not  a  firm  commitment  by 
either  party  for  developmental  testing,  but  is  preliminary 
notification  that  developmental  testing  may  be  required  at  some 
point  in  the  future. 
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b.  The  forecetet  permits  the  Army  to  identify  future 
requirements  for  developmental  test  resources  and  provides  a 
quantitative  basis  for  test  priorities  and  allocation  of 
resources.  It  also  supports  requirements  for  facility 
development  or  upgrade,  instrumentation  development  and 
acquisition,  and  test  methodology  studies,  as  well  as 
justification  for  MCA  plans. 

c.  For  program  planning  forecasting,  future  testing 
requirements  are  generally  those  scheduled  to  occur  beyond  the 
next  180  days  and  cover  the  current  FY,  the  budget  FY,  and  the 
POM  years.  If  requirements  can  be  forecasted  beyond  the  POM 
years,  it  is  beneficial.  To  initiate  a  program  planning 
forecast,  the  materiel  developer  should  provide  the  information 
reflected  at  figure  3-1. 

(INSERT  FIGURE  3-1) 

d.  Developing/updating  of  the  program  planning  forecast  is 
essential  and  should  be  accomplished  throughout  the  acquisition 
cycle.  This  can  be  done  efficiently  and  effectively  by  an 
exchange  of  information  throughout  the  T&E  planning  process: 

(1)  As  early  in  the  acquisition  cycle  as  possible,  as  T&E 
requirements  are  being  considered  during  concept  exploration  and 
definition. 

(2)  During  the  preparation/review  of  the  TEMP. 

(3)  As  a  result  of  negotiations  at  TIWG  meetings. 

(4)  During  program  reviews,  test  coordination  meetings, 

etc. 

Major  updates  will  result  in  a  revised  budget  estimate  being 
provided  to  the  materiel  developer. 

3-33.  Firm  requests  for  testing  requirements 

a.  Firm  test  requests  should  be  submitted  as  early  as 
possible  to  allow  TECOM  to  plan,  coordinate,  and  schedule 
resources  and  ensure  that  required  safety,  security,  and 
environmental  concerns  have  been  properly  addressed  prior  to  the 
test.  If  the  requirement  was  previously  identified  via  a  program 
planning  forecast,  the  transition  to  a  firm  request  is 
accomplished  smoothly  and  efficiently  as  most  of  the  detail  has 
been  previously  provided. 
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b.  The  firm  test  request  should  include  the  information 
reflected  at  figure  3-2. 

(INSERT  FIGURE  3-2) 

c.  Additionally,  the  firm  test  request  should  include 
documentation  required  by  regulation  to  be  provided  prior  to 
conduct  of  developmental  testing.  This  documentation  includes 
Safety  Assessment  Report,  Security  Classification  Guide,  and 
environmental  documentation  (e.g.,  REC,  EIS,  EA) .  If  these 
documents  are  not  available  at  the  time  the  test  request  is 
submitted,  the  request  should  reflect  a  date  as  to  when  the 
documentation  will  be  provided. 

d.  Any  other  documentation  or  information  which  would  enhance 
TECOM's  understanding  of  the  test  effort  should  be  included. 

3-34.  Submissions 

Both  firm  requests  and  program  planning  forecasts  should  be 
submitted  to  the  Commander,  TECOM,  ATTN:  AMSTE-TA-0,  Aberdeen 
Proving  Ground,  MD  21005-5055.  Requests  may  also  be  provided 
via  e-mail  (amstetao@apg-9.army.mil)  or  facsimile  (DSN  298-9170 
or  Commercial  (410)  278-9170) . 
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Chapter  4 

DT&E  Reporting  and  Documentation 


Section  I 

Developmental  Test  Readiness  Review  (DTRR) 


4-1.  General  . 

DTRR  is  conducted  to  determine  if  the  materiel  system  is  ready 
for  developmental  testing.  As  a  minimum,  the  DTRR  is  conducted 
prior  to  PPQT  for  materiel  systems  or  the  Software  Qualification 
Test  (SQT)  for  information  systems  for  all  ACAT  I  and  ACAT  II 
programs . 

4-2.  DTRR  Working  Group 

The  DTRR  working  group,  whose  members  include  the  parent  TIWG 
members  plus  others  deemed  appropriate,  reviews  all  pre-start 
activities  and  reguirements  which  may  impact  the  execution  of  the 
test  as  planned  by  the  TIWG.  The  objective  of  the  review  is  to 
determine  what  actions  are  required  to  assure  that  resources, 
training,  and  test  hardware  will  be  in  place  to  support  the 
successful  conduct  of  the  test,  and  to  ensure  that  the  T&E 
planning,  documentation,  design  maturity/configuration,  and  data 
systems  have  been  adequately  addressed. 

a.  The  DTRR  working  group  is  composed  of  a  representative 
from  each  of  the  following: 


(1) 

(2) 

(3) 

(4) 

(5) 

Office. 

(6) 

(7) 

(8) 

(9) 


Materiel 

Materiel 

Materiel 

MANPRINT 

Materiel 


Developer  (Chair) . 

Developer's  Safety  Office. 

Developer's  ILS  Office, 
representative . 

Developer's  Product  Assurance  &  Testing 


Combat  Developer. 

Developmental  Tester. 

Operational  Tester. 

Developmental  Evaluator /Assessor . 
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(10) 

Operational 

Evaluator. 

(11) 

Logistician. 

(12) 

Trainer. 

The 

DTRR  working 

group  should  be  formed  for  all  ACAT  I  and 

ACAT  II  programs.  For  ACAT  III  and  ACAT  IV  programs, 
establishment  of  a  working  group  is  at  the  discretion  of  the 
materiel  developer  and  the  developmental  tester.  In  cases  where 
a  full  DTRR  is  not  conducted,  the  materiel  developer  should 
conduct  an  internal  DTRR  to  assure  that  the  item/system  can 
successfully  complete  the  planned  testing.  This  DTRR  should  be 
chaired  by  an  independent  organization  within  the  command. 

4-3 .  Procedures 

a.  The  chairperson,  after  initial  coordination  with  the 
membership,  notifies  and  provides  each  member  a  DTRR  package  and 
a  pre-DTRR  checklist  (see  Figure  4-1  at  the  end  of  this  chapter) . 
Notification  of  the  time  and  location  of  the  review  plus  the  DTRR 
package  should  be  provided  at  least  2  weeks  before  the  review  to 
allow  members  to  determine  if  representation  by  their 
organization  is  required  and  to  effect  preliminary  internal 
coordination.  Member  agencies  determine  the  extent  of  their 
representation.  Since  all  representatives  may  not  attend  each 
review,  the  chairperson  may  indicate  recommended  attendance. 

b.  As  applicable,  the  DTRR  package  consists  of  the  following 
documentation : 

(1)  A  TIWG  coordinated  TEMP. 

(2)  Developmental  lEPs  and  TDPs  and  operational  TEPs. 

(3)  Safety  Assessment  Report. 

(4)  Applicable  Environmental  Document. 

(5)  Current  test  hardware  configuration. 

(6)  RAM  Failure  Definition/Scoring  Criteria. 

(7)  A  statement  of  the  status  of  the  SSP. 

(8)  A  statement  of  the  status  of  New  Equipment  Training 

(NET)  . 

(9)  A  statement  of  the  status  of  MANPRINT. 
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(10)  A  statement  of  the  status  of  instrumentation  and  data 
collection  and  reduction  facilities. 

(11)  An  ISLMT  approved  ILSP. 

(12)  An  airworthiness  statement. 

(13)  A  statement  on  status  of  software. 

c.  After  coordination  with  all  participants,  the  DTRR  working 
group  will  convened  at  the  call  of  the  chairperson. 

d.  The  DTRR  working  group  makes  recommendations  regarding  all 
issues  regarding  T&E  planning.  Each  representative  has  the 
responsibility  to  advise  participating  members  in  test  matters 
considered  to  be  of  mutual  concern. 

e.  In  the  event  of  disagreement  among  the  members,  issues  are 
presented  to  the  chairperson  for  resolution  through  normal 
command/ staff  channels. 

f.  The  chairperson  provides  minutes  of  the  DTRR  which  include 
a  Developmental  Test  Readiness  Statement.  This  statement 
verifies  that  the  system  is  ready  for  developmental  testing,  or 
if  there  are  action  items  identified  during  the  review  that  must 
be  satisfied  before  test  can  begin,  the  minutes  will  identify 
such  actions.  The  materiel  developer  will  ensure  that  all 
requirements  are  satisfied  before  the  test  begins.  The  minutes 
including  all  recommendations,  issues,  and  required  actions  are 
distributed  to  each  DTRR  participant  10  working  days  after  the 
DTRR. 


Section  II 

Test  Incident  Reports  and  Related  Reports 


4-4 .  General 

Timely  reporting  of  test  results  is  essential  and  is  accomplished 
through  Test  Incident  Reports  (TIR)  as  well  as  the  formal  test 
reporting  procedures.  TIRs  are  prepared  by  the  test  organization 
(Government  or  contractor)  to  provide  the  results  of  any  incident 
occurring  during  testing.  In  response,  the  materiel  developer 
prepares  a  Corrective  Action  Report  (CAR)  for  all  critical  or 
major  TIRs  which  reflect  the  developer's  analysis  of  the  problem. 
Details  of  test  incidents  and  related  reporting  are  contained  in 
Part  One  of  this  pamphlet. 
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Section  III 

Independent  Evaluation/Assessment  Plan 


4-5.  General 

The  developmental  lEP/IAP  is  formulated  by  the  developmental 
evaluator/assessor  in  close  cooperation  with  the  TIWG  members  to 
ensure  the  intent  of  all  technical  parameters  is  reflected. 

4-6.  Contents 

The  lEP/IAP,  as  a  minimum,  contains  a  brief  system  description, 
the  technical  parameters  (both  critical  and  noncritical)  and 
criteria  for  the  evaluation  of  each  parameter;  the  approach  and 
methodology  for  evaluation;  data  requirements /sources;  a 
description  of  that  portion  of  the  evaluation  which  will  require 
data  from  sources  other  than  test,  and  identification  of  program 
constraints.  The  plan  must  address  system  performance,  RAM, 
vulnerability/survivability,  electronic  interoperability, 
transportability,  MANPRINT,  safety,  etc. 

4-7.  Technical  Parameters 

As  developmental  evaluation  issues  (parameters)  are 
satisfactorily  resolved,  they  are  retained  and  annotated  in  the 
lEP/IAP.  Those  requiring  more  evaluation  or  revalidation  are 
included  in  the  next  evaluation  activity,  and  new  issues  are 
added,  as  appropriate.  Data  source  matrices  and  test  documents 
(if  needed)  are  revised  accordingly.  The  approved  lEP/IAPs  are 
furnished  to  all  members  of  the  TIWG.  lEPs  and  associated  TDPs, 
which  are  submitted  to  organizations  external  to  the  Army  for 
review  and/or  approval,  are  forwarded  through  the  DUSA(OR) . 


Section  IV 

Developmental  Test  Design  Plan  (TDP) 


4-8.  General 

The  TDP  guides  the  development  of  data  required  for  the 
independent  evaluation/assessment.  It  is  prepared  by  the 
developmental  lE/IA  and  coordinated  with  the  TIWG  members.  The 
TDP  for  combined  tests  should  expand  on  both  the  developmental 
lEP  and  operational  TEP.  For  test  programs  being  assessed  by 
TECOM,  the  TDP  is  incorporated  into  the  lAP. 

4-9 .  Content 

a.  The  TDP  addresses  all  developmental  test  parameters  and 
reflects  all  program  constraints  (dollars,  test  quantities, 
schedules,  issues,  etc.).  Additionally,  the  TDP  must  spell  out 
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the  form  in  which  the  data  are  needed  and  the  accuracy  with  which 
they  must  be  measured. 

b.  The  TDP  must  be  adequate  to  permit  the  developmental 
tester  to  develop  a  DTP  in  a  timely  manner  to  permit  the 
development  of  test  facilities  with  no  proqram  delays.  It  must 
clearly  define  the  evaluator/assessor 's  requirements  for  data. 
Detailed  coordination  between  the  evaluator/ assessor  and  the 
(developmental  tester  is  necessary  throuqhout  the  process. 

c.  As  a  minimum,  the  TDP  will  contain  the  appropriate 
reliability  test  strategy,  sample  sizes,  design  of  tests/ 
experiments,  minimum  test  requirements  to  measure  performance 
specified,  requirements  for  data  and  the  process  by  which  the 
data  will  be  verified,  and  identify  tests  in  order  of  priority  to 
ensure  that  the  more  critical  data  are  generated  early. 


Section  V 

Developmental  Detailed  Test  Plan 


4-10.  General  .  ^ 

The  developmental  Detailed  Test  Plan  (DTP)  is  prepared  by  the 
developmental  test  activity.  It  is  derived  from  and  implements 
the  lEP/IAP  and  the  TDP,  and  provides  explicit  instructions  for 
the  conduct  of  developmental  tests  and  subtests. 

4-11.  Coordination  ,  •  j 

The  DTP  governs  test  control,  data  collection,  data  analysis,  and 
the  necessary  administrative  aspects  of  the  test  program.  The 
DTP  must  be  coordinated  with  the  appropriate  developmental  lE/IAs 
and  may  be  coordinated  with  TIWG  members  to  ensure  that  the 
evaluation/assessment  reflects  the  requirements  of  the  TEMP  and 
TDPs. 

4-12 .  Content  .  ^ ^  ^ 

As  a  minimum,  the  DTP  should  address  the  test  objective,  test 
concept,  system  description,  test  personnel  requirements,  test 
criteria,  test  schedule,  and  required  coordination.  Each  subtest 
should  be  addressed  separately,  stating  the  criteria  to  be 
addressed  by  the  subtest,  the  data  to  be  obtained  during  the 
test,  and  the  procedures  to  be  used.  These  should  be  described 
in  sufficient  detail  to  reflect  what  will  occur  during  the  test. 
Military  Standards  (MIL-STDS)  and  Test  Operating  Procedures 
(TOPS)  should  be  used,  if  possible,  and  referenced  in  the  DTP. 
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Section  VI 
Test  Report 


4-13.  General 

Contractor  and  Government  test  agencies  provide  to  TIWG  members 
and  the  decision  review  body  a  copy  of  the  TR  at  the  conclusion 
of  the  test.  For  extended  test  phases,  an  interim  TR  is 
submitted  at  least  annually.  Test  results  must  be  comprehensive 
and  complete  before  presentation  to  the  milestone  decision 
authority . 

4-14 .  Requirements 

As  a  minimum,  final  draft  TRs,  authenticated  by  the  test  agency 
head,  are  required  prior  to  decision  reviews.  This  is  in 
consonance  with  policy  regarding  other  documentation  supporting 
the  acquisition  of  a  weapons  system.  In  order  to  facilitate 
this,  the  PEG  chairs  a  T&E  review  30  days  prior  to  decision 
review.  The  purpose  is  to  review  the  adequacy  of  past  tests, 
test  results  and  evaluations,  planning  for  future  testing,  and 
the  modification  of  test  strategy  to  accommodate  the  evolving 
acquisition  strategy.  Issues  not  resolved  in  this  forum  will  be 
brought  to  the  attention  of  the  DUSA(OR) . 

4-15.  Content 

The  format  of  the  formal  TR  parallels  that  of  the  DTP.  An 
Executive  Digest  provides  a  summary  of  the  significant  findings, 
the  test  objectives  and  concept,  and  a  description  of  the  test 
item.  Subtest  results,  in  addition  to  the  objectives,  criteria, 
and  test  procedure,  include  test  findings  and  a  technical 
analysis.  Appendices  include  the  test  program  criteria  (from  the 
DTP),  and,  if  required,  lengthy  test  data  presented  as  tables, 
charts,  illustrations,  etc.  The  formal  test  report  may  include  a 
preliminary  determination  of  deficiencies,  shortcomings,  and 
suggested  improvements. 


Section  VII 

Developmental  Independent  Evaluation/Assessment  Report 


4-16.  General 

Developmental  lER/IARs  are  prepared  by  the  developmental 
independent  evaluator/assessor  and  updated  as  additional  data 
becomes  available.  Coordination  of  information  contained  in  the 
lER/IAR  between  the  testers  and  evaluators  is  effected  to  ensure 
that  test  data  are  accurately  reflected  in  the  lER/IAR.  This 
updating  process  assures  the  decision  authority  has  the  latest 
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evaluation  results.  It  is  the  ’'continuous"  in  CE.  This  process 
highlights  those  issues  that  have  been  answered  and  require 
little  or  no  additional  analysis  as  well  as  those  needing  further 
evaluation. 

4-17.  Requirements 

a.  The  lER/IAR  addresses  both  the  critical  technical 
parameters  identified  in  the  lEP/IAP  and  other  issues  which  are 
appropriate  to  the  specific  item.  It  defines  the ^ methodology 
used  to  characterize  materiel  performance  (effectiveness,  RAM, 
survivability,  mobility/  transportability) ,  logistics,  MANPRINT, 
and  safety. 

b.  As  a  general  rule,  developmental  lER/IARs  are  prepared 
prior  to  MS  decision  reviews.  However,  for  very  small  programs, 
such  as  a  simple  modification,  an  expanded  TR  may  suffice.  This 
determination  will  be  made  jointly  by  the  developmental  tester 
and  the  developmental  independent  evaluator/assessor . 

4-18.  Content 

lER/IARs  contain,  as  a  minimum,  background  and  system 
description,  the  evaluation/assessment  of  each  technical 
parameter  (both  critical  and  noncritical) ,  the  safety 
confirmation,  and  conclusions  and  recommendations. 


Section  VIII 

Outline  Test  Plan  (OTP) 


4-19.  General  ^ 

The  OTP,  as  defined  in  Part  Five,  is  required  to  obtain  OT 
resources;  however,  if  DT  requires  the  use  of  user  troops  beyond 
resources,  the  developmental  tester  also  prepares  a  draft  OTP  to 
obtain  troop  and  equipment  support.  This  will  permit  early 
planning  for  the  resources  to  be  provided  through  the  Test 
Schedule  and  Review  Committee  (TSARC)  process  (e.g.,  U.S.  Army 
Forces  Command  (FORSCOM) ,  U.S.  Army  Training  and  Doctrine 
Command,  U.S.  Army  Pacific  Command)  and  for  resource  support 
through  the  FYTP  process.  (See  AR  15-38.) 


Section  IX 

Test  &  Evaluation  Master  Plan  (TEMP) 


4-20.  General 

The  developmental  testers  and  evaluators/assessors  are  primary 
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players  in  the  development  of  the  TEMP.  (For  additional 
guidelines  on  the  TEMP,  see  Part  Two.) 

4-21.  DT&E  Input 

a.  As  input  to  Part  I,  the  developmental  independent 
evaluator/  assessor,  in  coordination  with  the  combat  developer, 
the  materiel  developer,  and  the  developmental  tester,  develop  the 
critical  technical  parameters  (see  Section  XV) . 

b.  Both  the  evaluators/assessors  and  the  testers  provide 
support  to  the  materiel  developer  in  the  development  of  Part  II, 
the  Integrated  Test  Program  Summary. 

c.  Part  III,  Developmental  T&E  Outline,  which  provides  an 
overview  of  the  entire  DT&E  program,  is  developed  jointly  by  the 
developmental  tester  and  the  developmental  evaluator/assessor. 
This  section  of  the  TEMP  outlines  the  objectives  of  DT&E, 
identifies  the  DT&E  that  has  been  completed,  discusses  the  DT&E 
tq  be  conducted  with  emphasis  on  the  next  phase  of  the 
acquisition  cycle,  and  includes  a  description  of  LFTE,  if 
applicable. 

d.  The  developmental  tester  and  evaluator/assessor  have  a 
substantial  role  in  the  development  of  Part  V,  T&E  Resource 
Summary,  which  provides  a  summary  of  all  T&E  resources: 

(1)  Test  Articles.  The  developmental  evaluator/assessor, 
in  coordination  with  the  developmental  tester,  identifies  the 
number  of  test  articles  required  and  when  they  will  be  needed. 

(2)  Test  Sites  and  Instrumentation.  The  developmental 
tester,  with  support  from  the  evaluator /assessor,  identifies  the 
test  range/ facility  and  the  instrumentation  that  will  be  required 
for  each  test  to  be  conducted.  Details  on  acquiring  needed 
instrumentation  are  in  Part  Eight  of  this  pamphlet. 

(3)  Test  Support.  The  developmental  tester  determines 
what  support  equipment  is  required,  and,  if  not  available,  what 
must  be  acquired  for  the  specific  test  program. 

(4)  Threat  Systems/ Simulators.  The  developmental  tester 
assists  in  determining  the  requirements  for  and  availability  of 
current  assets  and  the  sufficiency  of  those  capabilities. 

(5)  Test  Targets  and  Expendables.  The  developmental 
tester  identifies  the  type,  number,  and  availability  requirements 
for  targets  and  other  expendables,  including  ammunition,  threat 
targets  for  lethality  testing,  and  threat  munitions  for 
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vulnerability  testing.  Details  for  obtaining  necessary  targets 
and  threat  simulators  are  outlined  in  Part  Eight. 

(6)  Simulation,  Models,  and  Testbeds.  The  developmental 
independent  evaluator/assessor  identifies  the  system  simulations 
required  for  the  developmental  T&E  and  the  resources  required  to 
validate  and  certify  their  use. 

(7)  Special  Requirements.  The  developmental  evaluator/ 
assessor,  in  conjunction  with  the  developmental  tester, 
identifies  any  significant  non-  instrumentation  capabilities 
required  for  the  developmental  T&E. 

(8)  T&E  Funding  Requirements.  The  developmental  tester 
provides  an  estimate  of  the  funding  required  to  pay  direct  costs 
for  the  developmental  tests  required  to  be  conducted  at 
government  ranges  and  test  facilities. 

(9)  Manpower /Personnel  Training.  The  developmental  tester 
identifies  the  key  manpower /personnel  and  training  required  for 
developmental  T&E.  These  requirements  must  be  identified,  to  tne 
degree  known,  at  Milestone  I. 


Section  X 

Summary  of  DT&E  Documents 


4-22.  General  ,  .  .. 

There  are  many  documents  required  to  plan  and  report  on  the  T&E 
that  takes  place  during  the  JLife  cycle  of  a  system.  To  provide  a 
summary  of  the  specific  documents  required  within  the  DT&E  arena, 
the  following  information  provides  a  brief  description,  source 
references,  and  the  activity  with  primary  responsibility  for  the 
document . 


4-23.  DT&E  planning  documents 
a.  T&E  Master  Plan 

(1)  Reference:  DODI  5000.2,  DOD  5000. 2-M,  AR  73-1. 

(2)  Responsibility:  Materiel  Developer 

(3)  Summary:  The  T&E  Master  Plan  is  the  basic  planning 
document  for  all  life  cycle  T&E  related  to  a  particular  system 
acquisition  and  is  used  in  planning,  reviewing,  and  approving  T&E 
activities,  and  must  be  approved  prior  to  the  start  of  any 
testing.  The  T&E  Master  Plan  addresses  the  T&E  to  be 
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accomplished  in  each  phase  of  the  life  cycle.  (See  Part  Two  of 
this  pamphlet.) 

b.  Developmental  Independent  Evaluation/ Assessment  Plan 
(lEP/IAP) . 

(1)  Reference:  AR  73-1. 

(2)  Responsibility;  Developmental  lE/IA.  The  lEP/IAPs 
are  prepared  in  close  coordination  with  the  TIWG  members. 

(3)  Summary;  The  developmental  evaluator/assessor 
prepares  an  lEP/IAP  which  details  all  aspects  of  developmental 
evaluation  responsibilities  relative  to  the  system  throughout  its 
acquisition  cycle.  The  lEP/IAP  supports  the  TEMP  by  addressing 
the  issues  for  testing;  describing  evaluation  of  issues  which 
require  data  from  sources  other  than  tests;  stating  the  technical 
parameters;  identifying  data  sources;  providing  the  approach  to 
the  evaluation;  and  identifying  .program  constraints. 

c.  Developmental  Test  Design  Plan  (TDP) . 

(1)  Reference:  AR  73-1. 

(2)  Responsibility;  Developmental  lE/IA.  Developmental 
TDPs  are  coordinated  with  TIWG  members. 

(3)  Summary:  The  TDP  is  responsive  to  the  technical 
parameters.  It  includes  a  complete  test  design;  a  description  of 
required  tests;  the  conditions  under  which  the  system  will  be 
tested;  and  a  statement  of  test  criteria  and  test  methodology. 

The  TDP  also  specifies  data  requirements  and  includes  plans  for 
data  collection  and  analysis.  Developmental  TDPs  will  address 
government  and  may  address  contractor  testing/modeling/simulation 
efforts,  if  appropriate.  For  assessed  programs,  the  TDP  is 
included  in  the  lAP. 

d.  Outline  Test  Plan  (OTP) . 

(1)  Reference;  AR  15-38  and  AR  73-1. 

(2)  Responsibility;  Usually  the  OT;  however,  if 
supplementary  operational  troops  are  required  for  DT,  the 
developmental  tester  also  prepares  an  OTP. 

(3)  Summary:  The  OTP  is  a  formal  resource  document 
submitted  for  the  TSARC  review.  All  acquisition  programs  must 
have  an  Army-approved  TEMP  prior  to  competing  in  the  TSARC 
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process  for  resources  and  commitments  to  provide  such  resources. 
The  document  contains  a  listing  of  the  resources  required  and  the 
administrative  information  necessary  to  support  the  test.  The 
document  also  contains  the  critical  test  issues,  test  conditions, 
a  brief  scope,  suspense  dates,  test  milestones,  and  cost 
estimates. 

e.  Five  Year  Test  Program  (FYTP) . 

(1)  Reference:  AR  15-38  and  AR  73-1. 

(2)  Responsibility:  OPTEC. 

(3)  Summary:  The  FYTP  is  a  compendium  of  OTPs  approved  by 

HQDA  (DCSOPS)  for  the  Chief  of  Staff,  U.S.  Army.  The  document 
identifies  validated  requirements  to  support  operational  tests  as 
well  as  those  DTs  for  which  operational  troops  are  required.  It 
is  a  tasking  document  for  the  current  and  budget  years  and 
provides  test  planning  guidelines  for  the  out  years. 

f.  Detailed  Test  Plan  (DTP). 

(1)  Reference:  AR  73-1. 

(2)  Responsibility:  Organizations  responsible  for  conduct 

of  test. 

(3)  Summary:  The  DTP  provides  explicit  instructions  for 
the  conduct  of  tests  and  subtests.  It  is  derived  from  and 
implements  the  TDP.  The  DTP  governs  test  control,  data 
collection,  data  analysis,  and  the  necessary  administrative 
aspects  of  the  test  program.  There  may  be  one  or  several  DTPS 
depending  on  the  complexity  of  the  program  and  the  number  of  test 
sites  or  test  organizations  involved  in  providing  data.  The  DTP 
must  be  coordinated  with  appropriate  lE/IAs  and  may  be 
coordinated  with  other  TIWG  members  to  ensure  that  the 
evaluation/assessment  reflects  the  requirements  of  the  TEMP  and 
TDPs.  DTPS  for  LFTE  are  approved  by  the  DUSA(OR) . 

4-24.  DT&E  Reporting  Documents 

a.  Test  Incident  Report  (TIR) . 

(1)  Reference:  AR  73-1  and  AR  702-3. 

(2)  Responsibility:  Organization  responsible  for  conduct 
of  tests. 
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(3)  Summary:  TIRs  are  used  as  the  medium  to  provide  the 
results  of  any  incident  occurring  during  test,  report  the  results 
of  subtests,  and  serve  as  interim  reports. 

b.  Corrective  Action  Report  (CAR) . 

(1)  Reference:  AR  702-3  and  AR  73-1. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  The  CAR  reflects  the  developer's  analysis  of 
the  problem  reported  by  the  TIR  and  the  status/description  of 
corrective  action  or  report  that  no  corrective  action  is 
proposed.  CARs  are  prepared  for  major  and  critical  TIRs. 

c.  Test  Report  (TR)  and  Expanded  TR. 

(1)  Reference:  AR  73-1. 

(2)  Responsibility:  Organization  responsible  for  conduct 

of  tests. 

(3)  Summary:  The  TR  is  a  formal  document  of  record  which 
reports  the  data  and  information  obtained  from  the  conduct  of 
test  and  describes  the  conditions  which  actually  prevailed  during 
test  execution  and  data  collection.  For  ACAT  IV  systems  which  do 
not  have  DOT&E  oversight,  an  expanded  TR  may  be  written.  An 
expanded  TR  is  a  test  report  with  evaluative  content  which  is 
endorsed  by  the  evaluator  in  lieu  of  a  separate  evaluation. 

e.  Developmental  Independent/Assessment  Evaluation  Report 
(lER/IAR) . 

(1)  Reference:  AR  73-1. 

(2)  Responsibility:  Developmental  lE/IA. 

(3)  Summary:  The  lER/IAR  provides  the  independent 
evaluation  of  the  system  and  is  based  on  test  data,  reports, 
studies,  simulations,  and  other  appropriate  sources.  It  also 
contains  the  independent  evaluator's  assessment  of  the 
parameters,  conclusions,  and  position  on  the  future  capability  of 
the  system  to  fulfill  the  approved  requirements.  The  lER/IAR 
will  contain  an  assessment  of  the  adequacy  of  testing  and  the 
need  for  additional  testing,  and  will  identify  program 
constraints  and  their  impact  on  the  evaluation. 

4-25.  Supporting  documents 
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a.  Record  of  Environmental  Consideration  (REC) . 

(1)  Reference:  AR  200-2. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  Briefly  describes  a  proposed  action  and 
contains  a  checklist  explaining  why  further  analysis  is  not 
necessary.  It  is  used  when  a  categorical  exclusion  applies  and 
there  is  no  measurable  impact  to  the  environment. 

b.  Environmental  Assessment  (EA) . 

(1)  Reference:  AR  200-2. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  Addresses  new  and  continuing  activities 
where  the  potential  exists  for  measurable  degradation  of 
environmental  quality.  This  document  concludes  with  either  a 
finding  of  no  significant  impact  or  a  statement  that  an 
environmental  impact  statement  is  necessary. 

c.  Environmental  Impact  Statement  (EIS) . 

(1)  Reference:  AR  200-2. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  If  the  EA  shows  that  the  system  will  impact 
the  environment  adversely,  or  is  controversial,  an  EIS  is 
prepared.  It  provides  full  discussion  to  the  public  on  all 
issues  associated  with  a  Federal  action  that  has  the  potential  to 
significantly  impact  the  natural  environment.  If  required, 
testing  is  performed  to  identify  and  quantify  the  environmental 
quality  issues. 

d.  Health  Hazard  Assessment  Report  (HHAR) . 

(1)  Reference:  AR  40-10. 

(2)  Responsibility:  Materiel  Developer. 

(3)  The  HHAR  is  the  formal  document  used  to  provide  an 
analysis  and  assessment  of  health  hazard  issues.  It  also 
provides  recommendations  for  eliminating  or  controlling  hazards. 
It  is  required  for  the  development  of  the  Safety  Assessment 
Report . 
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e.  Safety  Assessment  Report  (SAR) . 

(1)  Reference:  AR  385-16,  AR  40-10,  and  AR  73-1. 

(2)  Responsibility:  .  Materiel  Developer. 

(3)  Summary:  The  SAR  contains  data  and  information 
relative  to  personnel  and  equipment  hazards  inherent  in  the 
system  and  any  associated  operation  and  maintenance  hazards.  An 
assessment  of  the  data  will  be  provided  and  will  be  the  basis  for 
obtaining  a  Safety  Release.  Government  system  level  testing 
cannot  begin  until  the  SAR  is  received,  reviewed,  and  accepted  by 
the  test  organization. 

f.  Safety  Release  (SR). 

(1)  Reference:  AR  73-1  and  AR  385-16. 

(2)  Responsibility:  TECOM/HSC/MRDC/ISC. 

(3)  Summary:  The  SR  is  required  before  involving  soldiers 
in  testing.  The  SR  documents  the  precautions  that  must  be  taken 
by  the  soldier  to  avoid  system  damage  and  personal  injury.  The 
release  is  based  on  the  results  of  DT  and  data  presented  in  the 
SAR. 


g.  Safety  Confirmation. 

(1)  Reference:  AR  73-1  and  AR  385-16. 

(2)  Responsibility:  TECOM. 

(3)  Summary:  The  Safety  Confirmation  provides  the  safety 
findings  and  conclusions  and  states  where  the  specified  safety 
requirements/specifications  were  met.  The  Safety  Confirmation  is 
included  in  the  developmental  lER/IAR. 

h.  Human  Factors  Engineering  Assessment  (HFEA) . 

(1)  Reference:  AR  602-1. 

(2)  Responsibility:  AMC. 

(3)  Summary:  The  HFEA  summarizes  the  HFE  issues  based  on 
the  results  of  human  engineering  analyses,  system  testing,  and 
evaluation.  The  T&E  input  should  be  in  the  HFE  design,  soldier- 
machine  interface,  system  safety,  methodology,  data,  and  reports 
areas. 
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i.  System  Safety  Program  Plan  (SSPP) . 

(1)  Reference:  AR  385-16  and  MIL-STD  882. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  Implements  system  safety  engineering  program 
that  will  assess  the  safety  of  the  system,  and  assures  that  the 
system  meets  the  user's  safety  requirements  and  regulatory  safety 
standards . 

j.  System  MANPRINT  Management  Plan  (SMMP) . 

(1)  Reference:  AR  602-2. 

(2)  Responsibility:  TRADOC  and  Materiel  Developer. 

(3)  Summary:  Summarizes  program/plan  to  address  MANPRINT 
concerns  throughout  the  MAP. 

k.  System  Support  Package  Components  List  (SSPCL) . 

(1)  Reference:  AR  700-127. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  A  list  of  the  components  in  the  SSP  provided 
to  the  TIWG/ILS  Management  Team  for  review  and  furnished  60  days 
before  the  test  begins. 

l.  Integrated  Logistics -Support  Plan  (ILSP) . 

(1)  Reference:  AR  700-127. 

(2)  Responsibility:  Materiel  Developer. 

(3)  Summary:  Outlines  the  entire  ILS  strategy  for  a 
materiel  system. 

m.  System  Training  Plan  (STRAP) . 

(1)  Reference:  AR  350-35. 

(2)  Responsibility:  TRADOC. 

(3)  Summary:  Outlines  training  strategy  for  a  developing 
system.  Sets  milestones  for  development  of  the  Training  Product. 
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n.  NET  Plan. 

(1)  Reference;  AR  350-35. 

(2)  Responsibility;  TRADOC/Materiel  Developer. 

(3)  Summary;  Sets  training  dates  for  test  player 
instructor  evaluation  and  test  personnel  and  training  strategy  to 
support  unit  fielding. 
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Chapter  5 

DT&E  Considerations 


Section  I 

Reliability,  Availability,  &  Maintainability  (RAM) 


5-1.  General  . 

The  operational  RAM  values  specified  by  the  requirements 
documents  are  translated  into  technical  requirements  by  the  RAM 
Rationale  Working  Group.  The  technical  RAM  requirements  are 
translated  into  DT&E  requirements  by  the  developmental 
independent  evaluator/assessor .  Plans  must  be  specifically 
oriented  to  provide  the  test  data  required  to  assess  the  probable 
achievement  of  the  RAM  values. 

5-2.  RAM  scoring  conference 

RAM  scoring  conferences  are  held  before  publication/release  of 
the  test  report  and  are  chaired  by  the  materiel  developer  for  DT 
and  by  the  operational  evaluator  for  operational  tests.  Voting 
members  are  the  operational  evaluator,  developmental  evaluator, 
materiel  developer,  and  combat  developer.  The  failure  assessment 
data  is  obtained  from  the  TIRs  and  CARs.  Decisions  are  made  by  a 
majority  vote  of  the  primary  spokespersons.  If  unresolved 
differences  exist,  the  dissenting  opinions  are  formally 
documented  in  the  conference  minutes.  In  cases  where  a  majority 
opinion  does  not  exist,  the  operational  evaluator  makes  the  final 
determination  of  incidents  scoring  for  OT  and  the  developmental 
evaluator,  for  DT.  See  Chapter  11,  Part  One,  for  more  details  on 
scoring  conferences. 

5-3.  RAM  assessment  conference 

RAM  assessment  conferences,  when  convened,  are  chaired  by  the 
operational  IE  and  are  held  before  release/  publication  of  the^ 
TR.  Attendees  at  this  conference  are  the  same ^ as  for  the  scoring 
conference.  This  conference  is  conducted  to  discuss  and^ 
establish  the  test  data  base,  the  procedures  to  be  used  in 
assessing  the  data,  and  the  demonstrated  RAM  estimates.  The 
assessment  conference  is  conducted  under  guidelines  similar  to 
those  used  for  scoring  conferences,  except  there  is  no  tie 
breaking  vote.  Any  changes  to  the  test  data  base  must  be  by 
majority  opinion.  If  there  is  no  majority  opinion,  each  member 
reports  his/her  own  assessment.  Minutes  of  the  conference  are 
provided  to  all  attendees.  RAM  assessment  conference  procedures 
are  provided  in  Chapter  11,  Part  One. 
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Section  II 

Electromagnetic  Environmental  Effects  (E3) 


5-4 .  Test  and  evaluation  of  E3 

To  ensure  that  Army  materiel  is  in  compliance  with  E3  policy, 
testing  under  the  purview  of  an  Army  tester  and  an  independent 
evaluator /assessor  will  be  conducted.  Evaluations  will  assess 
the  probable  inter-  and  intra-system  E3  hardness,  as  well  as 
provide  guidance  and  theoretical  pretest  predictions.  Early  DT&E 
planning  will  ensure  the  use  of  currently  scheduled  tests  to 
fully  assess  the  E3  criteria  rather  than  requiring  new  or 
increased  testing.  See  Part  One  for  further  details  on  E3. 


Section  III 

Significant  Impact  Tests 


5-5.  General 

Significant  impact  tests  or  demonstrations,  i.e.,  those  involving 
multiple  participants,  multinational  involvement,  and  those  with 
potential  multinational  impact  require  careful  planning, 
staffing,  coordination,  and  approval.  These  events  require 
detailed  attention  to  the  technical  aspects  and  performance  of 
the  tests  and  demonstrations  and  the  early  involvement  of  policy 
makers . 

5-6.  Coordination 

Prior  to  the  announcement  of  initiation  of  significant  impact 
tests,  coordination  must  be  effected  with  both  the  DUSA(OR)  and 
the  Army  Acquisition  Executive  (AAE) .  In  this  way,  it  is  ensured 
that  Army  policy  makers  are  allowed  to  review  and  approve  the 
planning  to  include  public  affairs  or  Congressional  notification 
and  news  media  planning. 


Section  IV 

Manpower  and  Personnel  Integration  (MANPRINT)  (AR  602-2) 


5-7.  General 

Throughout  the  acquisition  process,  MANPRINT  will  be  a  factor  in 
all  T&E  planning.  Developmental  testing  will  be  planned  so  as  to 
provide  data  for  the  assessment  of  issues  regarding  the 
integration  of  all  MANPRINT  domains,  i.e.,  human  factors 
engineering,  manpower,  training,  system  safety,  and  health 
hazards.  This  assessment  will  determine  if  the  item  can  be 
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adequately  operated  and  maintained  by  soldiers  representative  of 
the  target  users,  with  the  proposed  system  training,  and  under 
the  expected  environment. 


Section  V 

Logistic  Supportability  (AR  700-127) 


5-8.  General 

Evaluation  of  materiel  supportability  is  mandatory  during  both  DT 
and  OT.  The  scope  of  the  evaluation/assessment  varies  depending 
on  the  characteristics  of  the  system  and  where  the  program  is  in 
the  acquisition  cycle.  The  developmental  evaluator/  assessor,  in 
coordination  with  the  materiel  developer  and  combat  developer , 
will  establish  the  logistic  support  parameters  to  be  addressed 
during  test  as  well  as  the  scope  of  testing  required  in  each 
acquisition  phase.  The  System  Support  Packages  (SSP)  provided 
for  developmental  test  and  evaluation/assessment  will  represent 
the  logistic  support  system  that  will  be  provided  when  a  system 
is  deployed  in  the  field.  (See  Part  One  for  more  detailed 
information  on  SSPs.) 


Section  VI 

Transportability  (AR  70-44,  AR  70-47  and  AR  700-27) 


5-9.  General.  Transportability  refers  to  the  ability  of  a 
system  to  be  moved  by  towing,  self-propulsion,  or  by  carrier  via 
railways,  highway,  air,  waterway,  or  helicopter,  and  airdrop 
modes  of  transportation  utilizing  existing  or  planned 
equipment /containers.  Transportability  testing  is  accomplished 
to  support  the  assessment  efforts  of  the  Military  Traffic 
Management  Command's  (MTMC)  Transportation  Engineering  Agency  and 
to  obtain  a  transportability  approval  from  MTMC.  This  testing 
also  supports  certification  for  external  air  transport  and 
airdrop. 


Section  VII 

Health  Hazard  Assessment  (AR  40-10) 


5-10.  General 

Developmental  testing  provides  data  regarding  personnel  health 
hazards  inherent  in  the  operation  and  maintenance  of  the  system. 
Planning  for  this  testing  must  be  considered  early  in  the  cycle 
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and  continues  throughout  the  acquisition  process.  Special 
attention  is  given  to  verifying  the  adequacy  of  safety  and 
warning  devices  and  any  other  measures  to  control  hazards.  An 
HHA  report  is  developed  by  the  U.S.  Army  Environmental  Hygiene 
Agency  from  data  gathered  from  a  variety  of  sources  and  includes 
the  results  of  developmental  tests  and  operational  tests  to  date. 
The  HHA  report  is  required  by  the  materiel  developer  for 
preparation  of 'the  Safety  Assessment  Report.  HHA  issues  are 
addressed  in  the  developmental  lEP/IAPs,  TPs,  and  lER/IARs. 


Section  VIII 

System  Safety  Testing  (AR  385-16) 


5-11.  General 

One  of  the  most  important  objectives  of  developmental  testing  is 
verifying  the  elimination  or  control  of  safety  and  health 
hazards.  The  developmental  tester  must  review  the  provisions  of 
MIL-STD  882  when  formulating  the  testing  program,  determining  the 
operational  environment,  and  establishing  operator  limits. 

5-12.  Safety  Assessment  Report 

a.  Prior  to  developmental  testing,  a  Safety  Assessment  Report 
(SAR)  is  prepared  by  the  materiel  developer.  The  SAR  is  the 
formal,  comprehensive  safety  report  which  summarizes  the  safety 
data  that  have  been  collected  and  evaluated  thus  far.  It 
expresses  the  considered  judgment  of  the  contractor  or  developing 
agency  regarding  the  hazard  potential  of  the  item  and  any  actions 
or  precautions  that  are  recommended  to  minimize  these  hazards  and 
to  reduce  the  exposure  of  personnel  and  equipment  to  them. 

b.  The  SAR  is  provided  by  the  materiel  developer  to  the 
combat  developer,  OT  agency,  and  DT  agency  at  least  60  days 
before  start  of  their  respective  tests.  Government  DT  will  not 
begin  until  an  SAR  has  been  received,  reviewed,  and  accepted  by 
the  test  agency.  The  test  agencies  — 

(1)  Use  the  SAR  information  to  integrate  system  safety 
into  test  planning  and  into  procedures  and  for  shipping  and 
handling  of  the  system. 

(2)  Ensure  that  DT  does  not  begin  until  an  SAR  has  been 
received,  reviewed,  and  accepted  by  the  test  agency. 

c.  The  SAR  format  is  provided  in  figure  5-1. 

(INSERT  FIGURE  5-1) 
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5-13.  Safety  Testing 

Developmental  testing  for  safety  is  characterized  by  systematic 
testing  of  materiel  using  highly  technical  equipment  and 
instrumentation  under  laboratory  or  other  rigorously  controlled 
conditions.  The  tester  obtains  the  hazard  tracking  list  (see  DA 
Pam  385-16  for  guidance  on  structure  and  procedures  for  hazard 
tracking)  before  starting  developmental  testing.  This  list  is 
used  with  the  SAR  to  identify  the  remedies  that  have  been  applied 
to  correct  previously  identified  hazards.  Safety  tests  are  then 
performed  to  verify  the  adequacy  of  the  remedy.  Specific  safety 
tests  are  also  performed  on  critical  devices  or  components  to 
determine  the  nature  and  extent  of  hazards  presented  by  the 
materiel.  All  safety  testing  will  be  conducted  according  to  the 
appropriate  test  operating  procedures/  international  test 
operating  procedures  (TOPs/ITOPs) ,  as  available.  Use  of  standard 
test  procedures,  as  developed  in  TOPs/ITOPs  ensures  usability  and 
adequacy  of  the  test  data  in  addressing  the  safety  test 
objectives. 

5-14.  Safety  Release 

No  testing  (developmental  or  operational)  involving  troops  will 
begin  until  a  safety  release  has  been  issued  to  the  test 
organization.  For  operational  testing,  the  materiel  developer 
should  request  a  safety  release  as  soon  as  the  requirement  is 
known.  TECOM  is  responsible  for  issuing  all  safety  releases 
except  for  systems  being  developed  by  ISC,  HSC,  and  MRDC. 

a.  The  Safety  Release  is  a  formal  document  issued  to  a  test 
organization  before  any  hands-on  use  or  maintenance  by  troops. 

b.  The  Safety  Release  indicates  that  the  system  is  safe  for 
use  and  maintenance  by  typical  user  troops  and  describes  the 
specific  hazards  of  the  system  or  item  based  on  test  results, 
inspections,  and  system  safety  analyses.  Operational  limits  and 
precautions  are  included.  The  test  agency  uses  the  data  to 
integrate  safety  into  test  cpntrols  and  procedures  and  to 
determine  if  the  test  objectives  can  be  met  within  these  limits. 

c.  A  Limited  Safety  Release  can  be  issued  on  one  particular 
system. 

d.  A  Conditional  Safety  Release  is  issued  when  further  safety 
data  are  pending;  for  example,  when  all  safety  tests  have  not 
been  completed  and  certain  aspects  of  the  test  must  therefore  be 
restricted . 

e.  The  Safety  Release  is  in  the  format  shown  in  figure  5-2. 


/ 

( 
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(INSERT  FIGURE  5-2) 

5-15.  Safety  Confirmation 

Prior  to  an  MS  decision,  a  safety  confirmation  is  provided  to  the 
decision  maker  as  part  of  the  lER/IAR.  The  safety  confirmation 
evaluates  the  safety  findings,  states  whether  the  specified 
safety  requirements  have  been  met,  and  evaluates  the  risk  of 
proceeding  to  the  next  phase  of  the  acquisition  cycle.  The 
safety  confirmation  is  provided  by  the  government  developmental 
tester. 

5-16.  References 

Additional  details  pertaining  to  system  safety  are  contained  in 
AR  385-16  and  DA  Pam  385-16. 


Section  IX 

Use  of  Nontest  Personnel  and  Volumteers  in  Developmental  Testing 
(AR  70-25) 


5-17.  General 

The  safety  of  test  personnel  is  of  paramount  concern  during 
testing.  Test  designers  ensure  that  testers  are  protected  from 
risks  in  the  performance  of  their  testing  duties  by  scrutiny  of 
the  SAR  and  safety  release  and  review  and  approval  of  detailed 
test  plans.  AR  70-25,  Use  of  Volunteers  as  Subjects  of  Research, 
requires  review  of  test  plans  by  a  Human  Use  Committee  when: 
testing  involves  greater  than  minimal  risk  or  tests  are  being 
conducted  by  military  or  civilian  personnel  not  qualified  to  test 
by  duty  assignment  when  the  test  calls  specifically  for  such 
qualifications. 


Section  X 

Natural  Environmental  Testing  (AR  70-38) 


5-18.  General 

Environmental  testing  parameters  are  derived  from  the 
requirements  documents  and  are  tailored  to  each  specific  system 
(MIL-STD  810) .  Test  results  from  environmental  chamber  tests 
cannot  substitute  for  test  data  from  natural  environment  tests; 
however,  the  use  of  chambers  can  be  an  effective  screening  early 
in  the  development  of  the  item. 

5-19.  Climatic  Design  Types 

Developmental  test  programs  must  recognize  the  basic 
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environmental  condition  and  all  the  capabilities  and  limits  for 
which  the  system  is  designed.  As  a  minimum.  Army  weapons  systems 
are  designed  for  the  basic  climatic  design  type.  The  Army 
recognizes  four  climatic  design  types;  hot,  basic,  cold,  and 
severe  cold.  Generally,  all  Army  equipment  must  operate  in  at 
least  the  basic  climatic  design  type.  Potentially  dangerous 
items  (e.g.,  ammunition)  will  be  tested  for  safety  in  all 
climatic  design  types  despite  the  chance  of  them  being  used  in 
those  climates. 

5-20.  Basic  Climatic  Design  Type 

A  condensation  of  the  environment  descriptions  in  AR  70-38  for 
testing  in  the  basic  climatic  design  type  is  reflected  in  table 
5-1.  In  order  to  take  maximum  advantage  of  the  testing  season 
for  the  basic  climatic  design  type  at  the  Army's  natural 
environmental  test  centers,  the  following  must  be  considered: 

(INSERT  TABLE  5-1) 

a.  Basic  cold.  The  winter  testing  season  at  the  Cold  Regions 
Test  Center  (Alaska)  is  from  mid-October  through  mid-March.  Test 
hardware  must  be  delivered  by  the  beginning  of  the  test  season  (1 
Oct) .  Items  received  after  December  are  not  assured  of  an 
adequate  test  season. 

b.  Constant  and  variable  high  humidity.  There  are  two 
testing  seasons  for  the  Tropic  Test  Site  in  Panama.  The  wet 
testing  season  runs  from  Apri 1 -November .  A  drier  testing  season 
runs  from  December-March.  The  materiel  developer  should  plan  for 
a  test  of  several  months  for  tropic  testing  in  order  to  realize 
the  synergistic  effects  of  the  tropic  environment  throughout  both 
seasons. 

c.  Basic  Hot.  The  optimum  testing  season  at  Yuma  Proving 
Ground  for  these  effects  is  from  mid-May  through  mid-September. 
Test  hardware  must  be  delivered  by  the  beginning  of  the  test 
season.  Items  received  for  test  after  July  are  not  assured  of  an 
adequate  test  season. 

5-21.  Semi-Protected  Environments 

Information  systems  or  subsystems  which  will  be  operated  in  semi- 
protected  environments,  such  as  forward  area  command  centers,  are 
also  subject  to  these  provisions.  Information  systems  installed 
and  operated  in  a  protected  environment  do  not  come  under  these 
provisions;  however.  Environmental  Control  Units  supporting 
information  systems  requiring  a  special  environment  must  be 
tested  or  certified  by  the  providing  agency  to  ensure  capacity, 
suitability,  and  continued  support. 
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Section  XI 
Integrated  Testing 


5-22.  General 

The  integration  of  testing  requirements  (i.e.,  combined  or 
concurrent  developmental  testing/operational  testing)  mandates  a 
coordinated  effort  by  all  members  of  the  acquisition  community  to 
ensure  that  testing  is  optimized.  While  developmental  testing 
and  operational  testing  are  separate  activities  conducted  by 
different  test  communities,  they  interact  frequently  and  are 
complementary.  Each  provides  a  unique  perspective  on  a  program. 

5-23.  Test  Objectives 

Developmental  testing  and  operational  testing  are  normally 
conducted  with  some  degree  of  concurrency  creating  a  challenge  to 
the  testing  community  to  ensure  that  separate  and  different  test 
objectives  are  accomplished  without  duplication.  In  those 
instances  when  developmental  testing  and  operational  testing  can 
be  combined  to  save  resources,  the  separate  test  objectives  must 
not  be  compromised.  In  any  case,  separate  independent 
developmental  and  operational  evaluations  are  conducted. 


Section  XII 

Test  Data  Confirmation 


5-24.  General 

The  purpose  of  test  data  confirmation  is  to  ensure  the  widest 
possible  use  of  data.  The  TIWG  first  determines  whether  or  not  a 
need  exists  to  confirm  certain  test  data.  A  review  and 
assessment  is  performed  of  each  test  and  the  criticality  of  the 
use  of  the  data.  This  determines  which  tests  require 
confirmation  so  the  data  generated  can  be  used  for  evaluation 
purposes.  Test  data  confirmation  is  determined  by  the  TIWG. 

5-25.  Acceptability  of  data 

In  those  instances  when  a  particular  facility's  ability  to 
provide  acceptable  data  is  in  doubt,  the  Government  developmental 
tester,  the  materiel  developer,  and  the  independent 
evaluator /assessor,  if  appropriate,  inspect  the  facility  to 
verify  acceptability  of  data.  For  this  reason,  it  is  essential 
that  the  TIWG  review  and  coordinate  on  the  T&E  portion  of  the  RFP 
prior  to  its  issuance.  The  following  factors  are  considered  in 
determining  the  acceptability  of  the  test  data  that  will  be 
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generated : 

a.  Ranges,  courses,  test  apparatus,  and  support  equipment 
available  to  tester. 

b.  Laboratory  facilities,  instrumentation,  and  calibration 
available  to  tester. 

c.  Test  personnel  experience  and  expertise,  test  procedures, 
and  data  collection  and  reporting  procedures  used  by  tester. 

5-26.  Government  Monitoring 

In  those  instances  when  the  test  data  from  a  particular  source  or 
procedure  would  not  otherwise  be  acceptable,  the  evaluator  may 
require  data  be  validated  through  full  or  partial  monitoring  of 
the  testing  by  government  test  personnel. 

5-27.  Confirmation  Process 

Once  the  confirmation  process  has  been  established,  the  materiel 
developer  relies  upon  the  government  developmental  tester  to 
provide  assistance  in  contractual  proceedings.  Prior  to  bid 
solicitation,  the  materiel  developer; 

a.  Provides  the  T&E  portion  of  the  RFP  to  TIWG  members  for 
coordination  and  to  confirm  test  data  acceptability. 

b.  Provides  to  prospective  contractors,  in  the  RFP,  the 
option  of  using  government  test  services,  funded  directly  by  the 
materiel  developer.  This  provides  flexibility  to  the  contractors 
as  well  as  providing  the  TIWG  a  known  source  of  acceptable  data, 
should  other  sources  prove  unacceptable. 

5-28.  Contract  Requirements 

To  help  ensure  acceptability  of  test  data,  contracts  specify  that 
the  contractor: 

a.  Provides  a  test  plan  to  the  materiel  developer  for  TIWG 
coordination  prior  to  testing. 

b.  Reports  test  incidents  to  the  materiel  developer  and 
evaluators. 

c.  Reports  the  corrective  actions  taken  in  response  to  test 
incidents  to  the  materiel  developer  and  evaluators. 

d.  Provides  a  test  report  to  the  materiel  developer  and 
evaluators.  (If  contractor  test  data  will  be  used  to  satisfy 
certain  technical  requirements,  a  copy  of  the  contractor  test 
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report  should  be  provided  to  the  government  developmental  tester 
by  the  materiel  developer.) 


Section  XIII 

Environmental  Impact  (AR  200-2) 


5-29.  General 

Formal  environmental  documentation  is  required  by  Congressional 
mandates  to  support  all  Federal  agency  actions.  Therefore,  prior 
to  the  initiation  of  any  testing,  environmental  documentation 
must  be  provided  by  the  materiel  developer  to  the  developmental 
tester  in  accordance  with  AR  200-2. 

5-30.  Categorical  Exclusions 

A  categorical  exclusion  allowed  by  AR  200-2  pertains  to 
developmental  and  operational  testing  on  a  military  installation 
where  tests  are  conducted  in  conjunction  with  normal  military 
training  or  force  maintenance  activities  producing  only 
incremental  impacts,  if  any,  and  provided  the  training/ force 
maintenance  activities  have  been  adequately  assessed,  where 
required,  in  other  Army  environmental  documents.  Although  this 
type  of  testing  is  categorically  excluded,  a  Record  of 
Environmental  Consideration  is  still  required. 

5-31.  Environmental  Documentation 

See  Section  XXXII,  Chapter  4,  for  the  three  levels  of 
environmental  documentation  which  can  be  submitted.  Detailed 
information  and  requirements  pertinent  to  environmental 
documentation  are  contained  in  AR  200-2. 
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Chapter  6 
Test  Technology 


6-1.  General 

Development  and  acquisition  of  test  technology  (test  methods  and 
instrumentation) ,  like  weapon  systems  development,  involve  an 
acquisition  strategy  and  require  necessary  lead  times  to  reach  an 
initial  operation  capability  (IOC) .  It  may  require  as  much  lead 
time  to  develop  the  test  instrumentation,  targets,  and  threat 
simulators  as  to  develop  the  weapon  system  it  will  test.  PM  ITTS 
(Instrumentation,  Targets,  and  Threat  Simulators)  is  responsible 
for  the  design,  development,  acquisition,  and  fielding  of  major 
instrumentation,  targets,  and  threat  simulators  (except 
strategic  defense  targets  which  are  developed  by  SDC) .  It  is 
important  to  have  the  early  involvement  of  PM  ITTS  to  effectively 
satisfy  user  needs,  especially  as  the  sophistication  of  the 
requirements  increases. 

6-2.  Test  technology  process 

Materiel  and  systems  being  developed  are  incorporating  more  and 
more  advanced  technology.  With  the  increased  complexity  and 
sophistication  of  the  systems,  the  testing  requirements  are  more 
stringent,  the  testing  problems  more  difficult  to  solve,  and  more 
time  is  needed  to  solve  those  problems.  If  the  development  of 
the  system  is  to  proceed  smoothly  and  in  a  timely  manner,  it  is 
imperative  that  test  technology  efforts  begin  prior  to  Milestone 
I  -  program  initiation.  Several  related  test  technology 
activities,  described  in  the  following  paragraphs,  need  to  be 
addressed  by  the  Army  test  community  as  early  in  the  acquisition 
cycle  as  possible. 

6-3.  Advanced  test  technology  concepts 

The  initial  effort  of  the  test  technology  process  involves  early 
identification  and  assessment  of  emerging  weapons  development 
technologies  as  a  basis  for  determining  future  test  technology 
requirements.  This  effort  should  be  initiated  with,  or  prior  to, 
technology  base  activities  and  involves  close  interaction  with 
Army  laboratories  and  development  commands.  Test  requirements 
(i.e.,  data  parameters  and  corresponding  data  accuracies)  must  be 
determined  and  compared  with  existing  capabilities  in  order  to 
identify  and  assess  test  deficiencies.  The  deficiencies  are 
provided  as  inputs  to  methodology,  instrumentation,  and  target 
development  programs  as  appropriate.  To  accomplish  this,  the 
network  of  Advanced  Systems  Concepts  Offices  at  research, 
development,  and  engineering  centers  interface  with  U.  S.  Army 
Laboratory  Command,  the  T&E  community,  and  the  PM  ITTS. 
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6-4.  Test  methodology 

Test  methodology  investigations  should  precede  instrumentation  or 
target  developments  and  identify  what  methods  or  techniques  are 
needed  to  properly  test  the  weapon  systems  or  materiel.  When 
appropriate,  the  testing  methods  might  be  established  as  standard 
test  procedures  so  the  results  of  tests  conducted  at  different^ 
times  or  places  can  be  compared  and  assessed.  The  identification 
of  needed  test  instrumentation  can  be  the  results  of  test 
methodology  investigations. 

6-5.  Instrumentation  development 

Instrumentation  development  is  necessary  only  when  existing 
instrumentation  within  the  Army  or  industry  cannot  collect  the 
required  data.  To  meet  the  testing  requirements,  existing  range 
instrumentation  might  be  modified  or  new  instrumentation 
developed.  The  modifications  and/or  developments  can  be 
accomplished  in-house  or  under  contract.  Refer  to  Part  Eight  of 
this  pamphlet  for  details. 

6-6.  Targets  and  threat  simulators 

The  successful  testing  of  weapon  system  is  dependent  not  only  on 
using  proper  test  instrumentation  but  also  on  whether  the  system 
is  tested  in  a  proper  threat  environment.  If  the  actual  threat 
system  is  not  available  to  support  required  testing,  the  use  of  a 
surrogate  target  or  threat  simulator  should  be  used.  The 
surrogate  target  or  threat  simulator  must  realistically  represent 
applicable  characteristics  of  the  actual  threat  system.  The 
degree  of  fidelity  required  will  change  depending  upon  the 
materiel  system  under  test  and  the  type  of  test  that  is  being 
conducated.  Targets  must  be  validated  as  properly  replicating 
the  threat  and  accredited  for  the  particular  test  in  which  they 
are  being  used.  Refer  to  Part  Eight  of  this  pamphlet  for 
details. 

6-7.  The  Army  Test  Facilities  Register . (TESTFACS) 

TESTFACS  identifies  and  describes  testing  capabilities  within  the 
U.S.  Army.  The  register  provides  information  about  major  test 
facilities  and  major  instrumentation  test  equipment.  Further 
information  regarding  TESTFACS  is  reflected  in  Part  One. 
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Chapter  1 
Introduction 


Section  I 
Overview 


1-1.  Purpose 

This  part  describes  the  methods  and  procedures  for  implementing 
current  regulations  and  directives  for  operational  test  and 
evaluation  (OT&E) .  It  also  describes  Continuous  Evaluation  (CE) 
by  the  operational  evaluator,  and  operational  testing  in  field 
experiments  in  support  of  the  combat  and  training  development^ 
process.  The  procedures  in  Part  Five  of  the  DA  Pamphlet  are  in 
accordance  with  AR  73-1,  Test  and  Evaluation  Policy,  30  November 
1990,  paragraph  1-4.  Information  in  this  part  applies  to  the 
U.S.  Army,  U.S.  Army  Reserve  and  the  U.S.  Army  National  Guard. 

1-2.  Philosophy  of  testing 

a.  Extent.  The  prerequisite  for  testing  is  based  on  the 
answer  to  the  question:  '*What  don't  we  know,  that  we  need  to  know 
that  can  be  found  out  only  by  testing?"  The  extent  to  which 
testing  is  conducted  must  provide  the  answer.  Although  the  time 
spent  is  only  a  small  fraction  of  the  complete  acquisition  cycle, 
the  influence  of  testing  is  significant.  Experience  has 
demonstrated  that  where  tests  have  been  eliminated  or  reduced, 
deficiencies  in  the  system  have  been  overlooked,  only  to  surface 
after  deployment;  resulting  in  expensive  and  time  consuming 
modifications.  Where  testing  has  been  adequate  and  complete, 
systems  have  gone  to  production  and  deployment  sooner  than 
anticipated,  thus  saving  time  and  money,  and  with  favorable 
results  reflected  in  the  field.  Unnecessary  duplication  of 
testing  efforts,  facilities,  or  programs,  however,  must  be 
avoided  and  is  a  specific  responsibility  of  all  concerned. 

b.  Principles.  Operational  test  and  evaluation  is  conducted 
in  keeping  with  principles  of  an  objective  test  and  an  impartial 
evaluation  to  provide  to  those  responsible  the  operational 
information  necessary  to  resolve  critical  issues.  Critical 
issues  are  those  questions  that  must  be  answered  to  enable  a 
decision  to  be  reached.  Adherence  to  these  principles  is 
necessary  to  assure  valid  estimates  of  a  system's  expected 
operational  effectiveness  (including  vulnerability)  and 
operational  suitability  (including  compatibility, 
interoperability,  reliability,  availability,  maintainability 
(RAM) ,  logistic  supportability ,  safety,  health,  human  factors, 
and  trainability) .  While  it  is  difficult  to  state  established 
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principles  simply  (and  experienced  personnel  would  differ  in 
their  choice  of  wording) ,  they  may  be  summarized  in  three  terms- 
adequacy,  quality,  and  credibility. 

(1)  Adequacy.  The  amount  of  data  and  realism  of  test 
conditions  must  be  sufficient  to  support  the  resolution  of  the 
critical  issues. 

(2)  Quality.  The  test  planning,  control  of  test  events, 
and  treatment  of  data  must  make  the  operational  information  clear 
and  accurate. 

(3)  Credibility.  Test  conduct  and  data  handling  must  be 
separated  from  external  influence  and  personal  self-interest. 


Section  II 
General  Procedures 


1-3.  General  policy 

a.  This  part  applies  to  all  materiel  systems  acquired  under 
the  auspices  of  the  AR  70  series  regulations,  information  mission 
area  (IMA)  systems  acquired  under  the  AR  25  series  regulations, 
and  IMA  systems  in  maintenance  (post  deployment  support) .  It 
also  applies  to  OT&E  of  combat  and  training  development  products 
developed  under  the  auspices  of  the  AR  71  series. 

b.  Acquisition  program  category  designations  of  materiel 
systems  are  explained  in  figure  1—1,  and  acquisition  program 
categories  of  IMA  systems  are  explained  in  figure  1-2. 

1-4 .  Overview  of  operational  test  and  evaluation  management 

a.  Policy.  For  OT&E  purposes,  acquisition  systems  are 
designated  as  either  evaluate  or  abbreviated  evaluate.  All 
systems  are  operationally  tested  and  evaluated  by  the  U.S.  Army 
Operational  Test  and  Evaluation  Command  (OPTEC) ,  with  the 
exception  of  those  specialized  systems  assigned  to  other  OT&E 
activities;  the  U.S.  Army  Health  Services  Command  (USAHSC)  for 
medical  materiel,  U.S.  Army  Intelligence  and  Security  Command 
(USAINSCOM) ,  and  Corps  of  Engineers  (COE).  (See  figure  1-3.) 
OPTEC 's  Operational  Evaluation  Command  (OEC)  performs  the  OPTEC 
evaluation  mission.  OPTEC 's  Test  and  Experimentation  Command 
(TEXCOM)  conducts  operational  and  other  tests  to  support  and 
assist  the  evaluation  function.  All  acquisition  category  (ACAT) 
I,  ACAT  II,  and  those  In-Process  Review  (IPR)/ACAT  III  and  IV 
systems  on  the  Director,  Operational  Test  and  Evaluation  (DOTE) 
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oversight  list  will  automatically  receive  the  full  level  of 
evaluation  effort  and  will  automatically  be  considered  as  "full 
evaluation”  systems  and  designated  as  "evaluate.”  The  remaining 
ACAT  III  and  IV  systems  will  be  designated  "abbreviated 
evaluate. " 

b.  Oversight.  Oversight  systems  are  of  special  interest  to 
the  Department  of  Defense  (DOD) ,  and  are  designated  by  the 
Director,  Operational  Test  and  Evaluation  (DOTE) .  Systems  of 
special  interest  to  the  Army  Acquisition  Executive,  Chief  of 
Staff  of  the  Army,  Vice  Chief  of  Staff  of  the  Army,  or  Deputy 
Undersecretary  of  the  Army  for  (Operations  Research)  (DUSA-OR) , 
are  designated  as  oversight  by  the  Office  of  the  Deputy  Chief  of 
Staff  for  Operations  and  Plans  (ODCSOPS) .  Commander,  OPTEC  or 
Commanders  of  other  OT&E  activities  (USAHSC,  USAINSCOM,  and  COE) 
can  elevate  abbreviated  evaluate  systems  to  "full  evaluation" 
status.  In  addition,  either  the  evaluator  or  tester  may 
recommend  elevation  of  a  system  to  the  Commander,  OPTEC  or  other 
OT&E  activity  due  to  the  system's  complexity,  the  testing 
involved,  procurement  risk,  doctrinal  change,  cost  or 
relationships  with  other  systems. 

c.  Evaluate.  All  ACAT  ID,  IC,  II,  (figure  1-1)  and  DOTE 
oversight  programs,  (corresponding  to  the  previous  category  of 
OTEA  evaluated) .  These  evaluated  systems  will  normally  have  the 
following  characteristic: 

(1)  Have  research,  development,  test,  and  evaluation  (RDTE) 
costs  of  more  than  $115  million  or  procurement  costs  of  more  than 
$540  million  (FY90  constant  dollars) . 

(2)  Are  technically  evaluated  by  the  Army  Materiel  System 
Analysis  Activity  (AMSAA) . 

(3)  Require  data  beyond  operational  test  (OT)  for 
operational  evaluation  purposes. 

d.  Abbreviated  evaluate.  Abbreviated  evaluate  incorporates 
both  the  old  categories  of  OTEA  Endorsed  and  Abbreviated 
Evaluation.  Systems  for  which  the  operational  test  (OT)  findings 
would  be  adequate  to  support  a  production  or  deployment  decision 
will  have  an  abbreviated  evaluation  by  the  operational  evaluator. 
In  these  cases  the  tester's  plan  contains  all  the  operational 
issues  needed  by  the  decision  maker.  The  operational  tester's 
report  contains  evaluative  findings  and  is  sanctioned  by  the 
operational  evaluator  before  going  to  the  decision  maker. 

e.  Documentation.  Documents  described  in  this  document,  such 
as.  Test  and  Evaluation  Plan  (TEP) ,  and  the  Test  and  Evaluation 
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Report  (TER)  are  the  only  documents  which  will  be  used  to  plan  or 
report  operational  test  and  evaluation. 

f.  Evaluation  procedures  for  evaluate  systems 

(1)  For  Milestone  I.  A  Materiel  Acquisition  Decision 
Process  (MADP)  position  consisting  of  an  Early  Operational 
Assessment  (EOA)  or  Abbreviated  Operational  Assessment  (AOA)  will 
be  prepared  by  the  operational  evaluator.  A  TEP  to  guide  the 
evaluation  between  Milestone  (MS)  0  and  MS  I  will  only  be 
required  if  actual  Early  User  Test  and  Evaluation  (EUTE)  is  to  be 
conducted  during  this  period. 

(2)  For  Milestone  II.  A  MADP  position  which  consists  of  a 
TEP  and  EOA  from  any  EUTE  conducted.  Even  if  EUTE  will  not  be 
conducted,  portions  of  a  TEP  will  be  required  for  the  phase  from 
MS  I  to  MS  II  to  guide  evaluation  or  assessment  and  to  identify 
data  sources  used  in  support  of  an  EOA  (e.g.,  developmental 
testing,  market  survey,  etc.). 

(3)  For  Milestone  III.  A  MADP  position  consisting  of  a  TER 
from  initial  operational  test  (lOT)  and  all  other  operational 
testing  conducted  prior  to  MS  III.  Multiple  tests  such  as,  lOT, 

Force  Development  Test  and  Experimentation  (FDTE) ,  and  limited 
user  test  (LUT)  may  be  planned  in  the  single  TEMP  document.  For 
intervening  milestones  (e.g.,  MS  Ilia)  supported  by  LUT  or  any 
other  less  than  full  operational  test,  an  operational  assessment 
(OA)  will  be  used  to  report  on  operational  effectiveness  and 
guitability  issues  resulting  from  these  tests.  If  an  Initial 
Operational  Test  and  Evaluation  (lOTE)  is  conducted  prior  to  the 
Low  Rate  Initial  Production  (LRIP)  release  decision,  then  a  TER 
will  be  written  to  report  on  the  findings  of  the  LRIP. 

(4)  Beyond  Milestone  III.  For  subsequent  milestones  and 

materiel  release  decisions.  The  Operational  Evaluator  (OEC,  or 
other  OT&E  activity)  provides  a  position  rele'  'e  to  materiel 
release  to  the  Materiel  Release  Board  (MRB)  c  sting  of  the 

most  recent  TER,  an  OA  or  AOA.  A  TER  will  be  ndatory  if  FOT  is 

conducted  equivalent  to  lOT.  An  OA  will  be  written  if  FOT  is 
conducted  equivalent  to  a  LUT  or  FDTE.  A  TEP  will  be  prepared  if 
FOT  or  significant  other  operational  testing  will  be  conducted 
subsequent  to  MS  III.  An  AOA  may  be  used  to  update  the 
evaluator's  position  for  subsequent  MRB. 

(5)  Reports.  Periodic  reports  may  be  rendered  on  evaluated 
systems  throughout  the  life  cycle.  Format  for  these  reports  will 
be  the  OA/EOA  or  AOA  at  the  discretion  of  the  evaluator  (subject 
to  guidance  from  Commander,  OPTEC  or  other  OT&E  activity,  or 
higher  authority) .  Frequency  and  timing  of  these  reports  will  be 
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based  on  requirements  of  DOD,  HQDA,  IPR  decision  authority. 
Commander,  OPTEC  or  other  OT&E  activity. 

1-5.  The  independent  operational  evaluation  (lOE)  process 
a.  Required  by  DODI  5000.2  and  AR  73-1  to: 

(1)  Assess  system  operational  effectiveness  and 
suitability. 

(2)  Address  readiness  to  proceed  to  the  next  acquisition 

phase . 


(3)  Assess  the  adequacy  of  testing  and  identifies  the 
unresolved  issues. 

(4)  Address  the  need  for  additional  testing. 

•  b.  An  independent  operational  evaluation  will  be  conducted  on 
all  ACAT  I,  II  and  ACAT  III  and  IV  designated  as  oversight 
systems.  (See  figure  1-4.)  They  are  planned  and  conducted  by 
the  operational  evaluator  (OE) .  The  OE  must  remain  independent 
from  the  combat  developer  (CBTDEV)  and  materiel  developer 
(MATDEV)  and  report  directly  to  the  decision  authority. 

(1)  The  evaluation  is  based  on  an  approved  set  of  critical 
operational  issues  and  criteria  (COIC) .  COIC  are  provided  by  the 
combat  developer  for  materiel  systems  or  the  functional  proponent 
for  IMA  systems,  prior  to  MS  I.  These  COIC  and  additional 
operational  issues  and  criteria  (AOIC)  are  documented  in  the  Test 
and  Evaluation  Plan  (TEP)  prepared  by  the  operational  evaluator 
and  operational  tester.  These  are  the  issues  which  the  test  and 
evaluation  must  answer  to  provide  information  for  a’  production 
and  deployment  decision.  (See  chapter  2,  this  part.) 

(2)  Operational  evaluations  may  be  identified  as, 
independent  evaluation,  operational  assessment,  early  operational 
assessment  or  abbreviated  operational  assessments.  One  system 
may  be  subjected  to  several  operational  evaluations  throughout 
its  life  cycle.  For  example,  prior  to  MSs  I  and  II  early 
operational  assessments  or  operational  assessments  predominate 
for  ACAT  I  and  II  systems.  MS  III  production  decisions  require 
an  independent  operational  evaluation  for  all  ACAT  I  and  II 
systems.  Abbreviated  operational  assessments  are  written  to 
support  milestone  decision  reviews  at  any  milestone  for  ACAT  III 
and  IV  systems.  After  MS  III,  the  requirement  for  an  independent 
operational  evaluation  for  ACAT  I  and  II  programs  will  be 
determined  by  the  scope  or  impact  of  changes. 
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(3)  For  IMA  programs  where  independent  operational 
evaluation  is  not  conducted  during  the  post  deployment  software 
support  phase,  the  tester  is  responsible  for  preparing  a  TEP  and 
performing  an  assessment  of  the  system  change  package  contents, 
if  such  testing  is  indicated  in  the  TEMP. 

c.  To  ensure  that  test  data  are  adequate  to  answer 
operational  issues,  the  evaluator  participates  in  test  planning, 
conduct,  reporting,  and  reviews  prior  tests  of  the  system  and 
other  data  sources  identified  in  the  TEP. ^  If  previous 
evaluations  have  surfaced  operational  deficiencies,  the  evaluator 
determines  whether  the  corrections  that  have  been  made  are 
effective.  Before  every  milestone  decision  review  (MDR) ,  the  lOE 
prepares  a  TER/OA/AOA  and  provides  it  to  the  decision  review 
body.  The  evaluator  uses  all  available  information  to  prepare 
the  official  position  for  the  milestone  decision. 

1-6.  Operational  and  other  user  testing  conducted  in  support  of 
the  materiel  acquisition  process 

a.  General 

(1)  OT  is  conducted  on  materiel  systems  with  typical  user 
troops  in  as  realistic  an  operational  environment  as  possible. 

OT  uses  personnel  with  the  same  military  skills  and  training  as 
those  who  will  operate,  maintain,  and  support  the  system  when  it 
is  deployed.  A  realistic  operational  environment  includes 
tactical  operations  conducted  in  accordance  with  the  system's 
wartime  operational  mode  summary/mission  profile  (OMS/MP) ,  which 
specifies  the  number,  type,  and  frequency  of  combat  operations 
during  a  period  of  time.  The  scenarios  used  in  OT  should  use  the 
tactics,  doctrine,  and  logistics  and  maintenance  support  concepts 
planned  for  use  when  the  system  is  fielded.  The  OT  threat 
represents  threat  systems'  capabilities  and  threat  tactics  and 
doctrine  postulated  at  postf ielding.  The  environment  for  these 
operations  may  include: 

(a)  The  employr.  of  opposing  forces. 

(b)  Electronic  and  other  enemy  countermeasures. 

(c)  Simulated  nuclear,  biological,  or  NBC  chemical 

warfare. 

(d)  Smoke  and  other  forms  of  battlefield  obscuration. 

(e)  Terrain  and  weather. 
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(2)  Operational  testing  can  provide  data  not  obtainable 
through  other  sources  (modeling  or  simulation)  or  may  be  used  to 
validate  previous  analytical  efforts.  It  is  required  , 

development  systems,  including  Nondevelopmental  Items  (NDI)  and 
Product  Improvements  (Pis) ,  unless  waived  or  not  required  by  the 
Test  and  Evaluation  Master  Plan  (TEMP)  or  the  approved 
Acquisition  Strategy  (AS) .  AR  73-1,  chapter  5,  discusses 
operational  testing  in  more  detail. 

b.  Early  user  test  and  experimentation  (EUTE) .  EUTE  is  a 
generic  term  encompassing  all  system  tests  employing  . 
representative  user  troops  during  the  Demonstration  Validation 
phase  prior  to  MS  II.  The  purpose  of  EUTE  is  to  test  materiel 
concepts,  support  training,  and  logistics  and 

interoperability  problems  and  future  testing  requirements.  EOT 
provides  data  for  an  early  operational  assessment  to  support  the 
MS  II  decision.  FDTE  and/or  CEP  may  comprise  all  or  part  or 
EUTE. 


(1)  Early  user  test  (EOT).  An  EUT  is  a  test  prior  to  MS  II 
conducted  with  RDTE  funds.  EUT  use  procedures  described  for  OT 
modified  as  necessary  by  maturity  and 

systems  and  support  packages.  EUT  seeks  answers  to  known  issues 
which  roust  be  addressed  prior  to  MS  II. 

(2)  Early  user  experiment  (EUE) .  An  EUE  is  a  field 
experiment  conducted  to  generate  data,  which  is 

to^identify  potential  systems  related  solutions,  and/or  to  define 
issues  to  be  addressed  at  MS  II  and  beyond. 


c.  Limited  user  test  (LUT) .  The  LUT  is  a  generic  term 
encompassing  all  RDTE  funded  testing,  normally  conducted  between 
MS  II  and  MS  III  that  is  not  a  part  of  lOT  as  defined  below. 
addresses  limited  operational  issues  and  is  used  to  accomplish 
the  following  objectives; 


fl)  Testing  necessary  to  supplement  developmental  testing 
prior  to  a  MS  Ilia  decision  to  purchase  long  lead  or  LRIP  items 
for  lOT. 


(2)  Testing  necessary  to  verify  a  fix  to  a  problem 
discovered  in  lOT  that  must  be  verified  prior  to  the  production 
decision  (i.e.,  problem  is  of  such  importance  that  verification 
of  fix  cannot  be  deferred  to  FOT) . 


(3)  As  needed  to  support  NDI  or  a  Materiel  Change  (MC) 
acquisitions  that  do  not  require  a  dedicated  phase  of  lOT  prior 
to  a  production  decision. 
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(4)  A  LUT  will  not  be  used  to  circumvent  requirements  for 
lOT  prior  to  a  production  approval  decision  as  prescribed  by 
statute,  DOD  directives  and  AR  73-1. 

(5)  A  LUT  will  not  be  used  to  piecemeal  lOT  through  a 
series  of  limited  objective  tests. 

d.  Initial  operational  test  and  evaluation  (lOTE) .  The 
results  of  EUTE  can  be  incorporated  into  lOTE.  lOTE  for 
development  systems  includes  all  system  components,  such  as 
hardware,  associated  support  packages,  ground  support,  computer 
software,  training,  test  measurement  and  diagnostic  equipment 
(TMDE) ,  and  all  systems  with  which  the  system  under  test  must 
operate.  Waiver  requests  for  lOTE  are  supported  by  plans  and 
schedules  for  obtaining  relevant  data  from  other  sources  and  are 
approved  at  MDRs.  lOTE  is  characterized  by: 

(1)  Use  of  production-representative  materiel  systems. 

(2)  Organizational  units,  table(s)  of  organization  and 
equipment  (TOE)  units,  provisional  units,  or  elements  typical  of 
those  that  will  employ  and  support  the  system. 

(3)  Employment  under  realistic  simulated  combat  conditions 
equivalent  to  those  expected  during  the  initial  operating 
capability  (IOC)  timeframe  and  against  the  threat  postulated  for 
the  system's  deployment.  The  threat  capabilities  are  normally 
representative  of  these  projected  for  IOC  plus  2  years. 

e.  Follow-on  operational  test  and  evaluation  (FOTE) . 

(1)  FOTE  is  conducted  after  the  decision  to  enter  the 
production  and  deployment  phase.  FOTE  is  conducted  to  ensure 
that  production  items  meet  operational  effectiveness  and 
suitability  requirements;  to  validate  corrections  to  identified 
sys+^em  to  validate  corrections  of  training  and  logistical 

de"  1» ncies;  and  to  resolve  issues  remaining  after  the 
fu  cale  production  decision.  FOTE  is  conducted  on  production 
itt  using  the  IOC  unit  as  a  normal  part  of  an  acquisition 
program. 

(2)  The  operational  evaluator  should  minimize  the  need  for 
FOTE  by  making  maximum  use  of  other  data  sources.  As  much  as 
possible,  FOTE  uses  current  and  complete  support  packages, 
organizational  structures,  doctrine  for  employment, 
supportability ,  threat,  communications  and  control,  tactics, 
training,  and  interfaces  with  other  systems. 
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(3)  An  FOT  may  be  conducted  in  the  same  manner  and  to  the 
same  depth  as  an  lOT.  An  FOT  may  be  conducted  for  limited 
objectives  in  the  same  manner  as  an  LUT.  An  FDTE  may  accomplish 
the  objectives  of  an  FOT.  The  evaluator  tailors  the  extent  of 
the  FOT  to  the  requirements. 


1-7.  Operational  and  other  user  testing  conducted  in  support  of 
the  information  mission  area  (IMA)  acquisition  process 


a.  Operational  testing.  Previously  called  Software  or 
System's  Acceptance  Test  (SAT)  or  similar  tests,  will  be 
conducted  in  a  realistic  operational  environment  using  troops  or 
assigned  civilians  from  representative  units  or  organizations, 
and  incorporating  the  approved  threat. 


b.  supplemental  site  test  (SST) ,  information  systems  (IS) . 

An  SST  may  be  necessary  for  those  systems  which  execute  in  multi¬ 
hardware  and  operating  system  environments.  The  SST  supplements 
the  lOT  SAT  and  its  conduct  similar  to  a  Lead  Site  Verification 

Test  (LSVT) . 

c.  Initial  operational  test  (lOT) .  Testing  for  a  MS  III 
decision  is  called  an  lOT.  Between  MS  III  and  system  retirement, 
testing  is  called  Post  Deployment  Software  Support  (PDSS)  for  IMA 
systems,  and  Follow-on  Operational  Testing  (FOT)  for  materiel 
systems.  LSVT  applies  only  to  emergency  or  urgent  date  driven 
changes. 


d.  Independent  evaluation.  Major  systems  (Class  I  through  IV 
systems  (AR  25-3  and  figure  1-2)  will  have  an  independent 
operational  and  developmental  evaluation.  The  assigned 
independent  technical  and  operational  evaluators  provide  the  only 
official  evaluation  report. 


e  See  chapters  5  and  6,  Part  One  for  a  discussion  of^ 
incremental  acquisition,  OT&E,  and  fielding  strategies  suitable 
for  IMA  systems. 

1-8.  Operational  test  and  experiments  conducted  in  support  of 
the  combat  and  training  development  process 


a.  Force  Development  Test  and  Experimentation  (FDTE).  FDp 
is  a  generic  term  encompassing  a  range  of  tests  and  experiments 
conducted  with  troops  under  field  conditions  to  support  both 
materiel  system  acquisition  and  the  development  of  doctrine, 
training,  organizations,  logistics,  materiel  concepts  and/or 
requirements.  In  support  of  materiel  acquisition,  FDTE 
support  definition  of  the  materiel  requirement  or  assess  the 
doctrine,  training,  organization,  and  logistics  developed  for  the 


1-9 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992  (DRAFT) 


materiel  system.  An  FDTE  also  supports  the  development  and 
approval  of  concepts  and  doctrine,  training,  and  organizations 
not  specifically  tied  to  a  materiel  system  acquisition.  Although 
there  are  no  absolute  requirements  for  either  FDE  or  FDT,  FDE 
will  normally  be  more  applicable  early  in  the  acquisition  process 
when  combat  and  training  developers  are  seeking  to  solidify  and 
mature  the  doctrine,  training,  organization,  and  logistics 
aspects  of  the  system.  FDTE  are  funded  by  TRADOC. 

(1)  Force  development  test  (FDT) .  An  FDT  is  conducted  to 
generate  data  which  will  be  used  to  evaluate  the  effectiveness 
and  suitability  of  doctrine,  training,  leader  development, 
organizations,  and  or  logistics,  either  related  to  a  materiel 
system  or  a  separate  development.  The  major  purpose  of  an  FDT  is 
to  determine  if  the  product  tested  meets  the  stated  requirement. 
Results  of  the  test  may  also  be  used  to  further  refine  the 
product  (e.g.,  doctrine  or  training).  An  FDT  will  normally  be 
scheduled  later  in  the  process,  especially  just  prior  to  an  lOT 
or  FOT,  to  ensure  these  aspects  are  ready  to  be  included  in  the 
operational  test  of  the  total  system.  FDT  conducted  following 
the  full  production  decision,  using  the  initial  operational 
capability  (IOC)  unit,  is  also  appropriate  to  address  open  issues 
subsequent  to  FOT.  FDTs  may  be  funded  by  either  OMA  or  RDTE. 

(2)  Force  development  experiment  (FDE) .  FDEs  are  conducted 
to  generate  data  which  will  be  used  to  identify  and  refine 
proposed  solutions  in  the  areas  of  doctrine,  training, 
organizations,  logistics,  as  well  as  materiel  requirements. 

FDEs  are  funded  by  TRADOC. 

b.  Concept  evaluation  program  (CEP) .  A  CEP  is  a  TRADOC 
program  to  provide  quick  reaction  and  innovative  evaluation  to 
resolve  combat  and  training  development  issues.  The  primary 
focus  is  on  development  of  a  materiel  requirement.  CEP  is  RDTE 
funded,  and  OPTEC  maintains  these  funds  for  disbursement  at  the 
direction  of  TRADOC.  TRADOC  conducts  a  CEP  Schedule  and  ’  /iew 
Committee  (CEPSARC) ,  similar  to  the  TSARC,  to  approve  anc 
prioritize  CEP  requirements.  As  with  FDTE,  planning  and 
execution  is  patterned  after  OT&E  of  materiel  systems,  wit  as 
much  scientific  rigor  as  practical.  Separate,  dedicated  tests 
may  be  necessary  to  provide  data  to  support  the  CEP  evaluation. 
These  are  called  CEP  tests  to  distinguish  them  from  other  tests 
conducted  for  customers. 

c.  Customer  test  (CT) .  A  test  conducted  by  OPTEC  for  a 
requesting  agency  external  to  OPTEC.  The  requesting  agency 
coordinates  support  requirements  and  provides  funds  and  guidance 
for  the  test.  These  test  are  not  directly  responsive  to  Army 
program  objectives  and  are  not  scheduled  or  approved  through  the 
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General  Officer  Test  Schedule  and  Review  Committee  (TSARC) 
process  unless  external  resources  are  required  for  test  support. 

1-9.  The  relationship  of  operational  testing  to  developmental 
testing,  contractor  testing,  and  live  fire  testing 

a.  OT  is  normally  conducted  separate  from  DT,  such  as; 
technical  feasibility  test  (TFT) ,  qualification  test  (QT) , 
preproduction  qualification  test  (PPQT) ,  production  qualification 
test  (PQT) ,  and  contractor  test.  (See  AR  73-1.)  DT  and  OT  may 
be  combined  or  conducted  concurrently  whenever  the  objectives  of 
both  OT  and  DT  can  be  met  and  whenever  clearly  identified, 
significant  cost  or  time  benefits  would  result.  When  combining 
DT  and  OT  care  must  be  taken  to  ensure  a  successful  DT  is  not  a 
criterion  for  a  successful  OT.  Operational  evaluation  reports 
are  always  published  independent  from  developmental  evaluation 
reports.  The  extent  to  which  DT  and  OT  are  either  combined  or 
concurrent  is  addressed  while  the  acquisition  strategy  is  being 
formulated  and  is  included  in  the  TEMP. 

(1)  A  combined  test  is  a  single  test  using  the  same 
materiel  and  users  in  support  of  both  the  developmental  and 
operational  evaluations. 

(2)  Concurrent  testing  consists  of  multiple  tests  using 
separate  materiel  and  users  and  which  are  conducted  at  the  same 
time  for  supporting  developmental  and  operational  evaluations. 

b.  Contractor  testing.  A  contractor  test  is  any  test 
conducted  by  the  materiel  development  contractor  under  contract 
to  the  materiel  developer  (Program  Manager) .  These  tests  are 
normally  conducted  prior  to  materiel  acceptance  by  the  PM. 
Contractor  tests  are  non-government  tests  integrated  into  the  T&E 
process  through  the  TEMP  to  provide  data  for  evaluation  purposes. 

c.  Live  fire  test  and  evaluation  (LFT&E)  results  will  be 
integrated  into  the  overall  evaluation  of  designated  systems. 

(See  Part  VI  of  this  pamphlet.)  The  objective  of  LFT&E  is  to 
provide  a  timely  and  thorough  assessment  of  the  vulnerability 
and/or  lethality  of  a  system  as  it  progresses  through  its 
development  and  subsequent  production  phases. 
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Figure  1-1.  Acquisition  categories  (materiel  systems) 
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III 

$50-$100  million 
or 

$15  million 

Total  program 
cost  or 

1  year 

Army  MAI SRC 

Class 

IV 

$10-$50  million 

Total  program 
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VI 

Under 
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Figure  1-2.  Acquisition  categories  (information  mission  area) 


EVALUATE  THE  OPERATIONAL  EFFECTIVENESS  AND  SUITABILITY  OF 
DEVELOPING  SYSTEMS  WHEN  USED; 

-  IN  A  REALISTIC,  OPERATIONAL  ENVIRONMENT 

-  IN  OPERATIONALLY  REALISTIC  SCENARIOS 

-  BY  TYPICAL  SOLDIERS  OR  ORGANIZATIONS 

-  ACCORDING  TO  APPROVED  TACTICS,  DOCTRINE,  AND  OPERATING 
PROCEDURES 


Figure  1-3.  OT&E  objectives 
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STARTS  AT  MILESTONE  0  AND  CONTINUES  THROUGH  MILESTONE  IV 

INTEGRAL  PART  OF  THE  ACQUISITION  STRATEGY  (AS) 

TO  MAKE  A  DIRECT  CONTRIBUTION  TO  THE  TIMELY  DEVELOPMENT, 
PRODUCTION,  AND  FIELDING  OF  SYSTEMS  WHICH  MEET  THE 
USER’S  REQUIREMENT  AND  ARE  EFFECTIVE,  SUITABLE,  AND  SAFE 

CONTRACTOR,  TECHNICAL,  AND  USER  TiE 


Figure  1-4.  Operational  evaluation 
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Chapter  2  .  •  ^.v. 

Continuous  Independent  Operational  Evaluation  in  the 

Acquisition  Process 


Section  I 
Introduction 


2-1.  Nature  of  Continuous  Evaluation  (CE) 

CE  begins  as  early  as  possible  in  the  life  cycle  of  system 
management.  As  necessary,  CE  is  conducted  throughout  the  system 
acquisition  process  to  assess  acquisition  risks,  to  evaluate  ^ 
operational  effectiveness  and  suitability,  to  evaluate  logistic 
and  training  supportability ,  and  to  determine  interoperability 
with  NATO  and  other  systems. 

2-2.  Life  cycle  testing  ,  •  ^  j  • 

Most  testing  commences  with  competitive  tests  to  validate  design 
concepts  for  selection  of  a  system  for  further  development. 

Tests  of  selected  foreign  systems  that  are  viable  alternatives 
are  conducted  by  US  or  joint  U.S./NATO  allies ^throughout  the 
acquisition  cycle,  as  appropriate.  When  feasible  and  practical, 
the  tests  are  conducted  with  representative  prototypes  in 
realistic  operating  environments.  When  tests  at  the  overall 
system  level  are  determined  to  be  infeasible  or  impractical, 
competitive  prototype  tests  of  critical  subsystems  are  considered 
in  the  same  manner  as  described  above. 

2-3.  Need  for  success  in  testing 

Tests  need  not  be  repeated  if  adequate  results  are  achieved. 
However,  if  test  results  reflect  significant  deficiencies,  the 
decision  review  will  not  permit  program  advancement  into  a 
succeeding  phase  until  those  deficiencies  have  been  corrected 
and,  if  necessary,  the  corrections  verified  in  a  retest.  A 
deficiency  will  be  considered  significant  if  it  would  make  the 
system  unacceptable  for  deployment,  or  if  correction  involves^ 
more  than  very  routine  engineering.  Included  in  this  definition 
are  major  inadequacies  in  support  and  test  equipment,  supply 
support,  transportation  and  handling,  technical  data,  facilities, 
and  personnel  and  training  (the  system  support  package) . 

2-4 .  Conduct  of  CE  ... 

The  conduct  of  CE  requires  active  participation  of  independent 
operational  evaluators  throughout  the  acquisition  process.  It 
requires  effective  interaction  with  an  exchange  of  ideas  and  data 
with  the  acquisition  community. 
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Section  II 
Objectives  of  CE 


2-5.  CE  integration 

Foster  the  integration  of  all  appropriate  test  and  analysis 
information  in  arriving  at  the  assessment  of  a  system's 
operational  effectiveness  and  suitability. 

2-6.  CE  opportunities 

Create  opportunities  for  early  operational  evaluator  involvement 
in  the  development  and  acquisition  process  in  order  to  avert 
operational  problems  as  the  system  matures. 

2-7.  CE  understanding 

Enhance  the  Army's  understanding  of  the  system's  readiness  for 
operational  test. 


Section  III 

Continuous  Evaluation  Principles 


2-8.  CE  definition 

CE  is  a  continuous  process  extending  from  concept  definition  into 
deployment  which  evaluates  the  operational  effectiveness  and 
suitability  of  a  system  by  analysis  of  all  available  data.  CE  is 
an  evaluation  methodology  for  ensuring  responsible,  timely,  and 
effective  evaluations  of  the  status  of  a  system  in  its  progress 
toward  mature  system  operational  effectiveness  and  suitability. 
The  independent  operational  evaluator  (lOE)  is  responsible  for 
performing  CE  on  all  major  and  selected  nonmajor  systems. 

2-9.  CE  background 

a.  The  CE  strategy  resulted  from  a  series  requisition 
decisions  which  were  adversely  impacted  by  th  ek  of  early 
testing  and  by  the  evaluation  of  immature  sys  '.  In  these 
evaluations,  the  levels  of  achieved  system  efft  :tiveness  were 
driven  by  deficiencies  in  hardware,  software,  training,  logistics 
support,  and  tactics.  Because  the  evaluation  strategy  did  not 
encourage  aggressive  early  identification  and  correction  of  these 
operational  deficiencies,  system  evaluations  in  support  of 
production  and  fielding  decisions  often  reflected  lower  than 
required  system  effectiveness  and  unsupported  predictions  of 
future  effectiveness  growth.  Decision  makers  had  to  make  their 
judgments  based  on  these  evaluations  without  the  assurance  that 
the  system  could  meet  required  levels  of  effectiveness  when 
deficiencies  were  corrected. 
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b.  The  Army  needed  a  more  effective  evaluation  strategy, 
requiring  more  active  independent  evaluator  involvement 
throughout,  earlier  operational  test  and  experimentation,  more 
effective  use  of  all  information  and  Army  analytic  resources,  and 
more  frequent  reporting  of  findings,  conclusions,  and  status 
reports.  This  improved  evaluation  strategy  would  help  to  ensure 
the  identification  and  correction  of  deficiencies  prior  to  the 
formal  operational  test  which  supported  the  production  decision. 
It  would  also  improve  the  evaluator's  ability  to  quantify  the 
effectiveness  of  mature  systems.  As  a  result,  the  Army  would  be 
less  likely  to  face  the  potential  loss  of  important  systems 
attributable  to  immature  development  or  the  developer's  failure 
to  anticipate  problems. 

2-10.  Scope  of  CE 

CE  encompasses  a  broad  analytical  approach  to  the  operational 
evaluation  of  a  materiel  acquisition  program  from  earliest 
concept  definition  into  deployment.  CE  has  evolved  to  include 
examination  of  source  selection,  Nondevelopmental  Item  (NDI) 
market  investigations,  materiel  change,  and  post-fielding  system 
effectiveness  to  provide  extensive  coverage  of  MAP  events.  This 
operational  evaluation  encompasses  the  assessment  of  requirements 
definition,  operational  concepts,  training  requirements,  and  life 
cycle  support  requirements.  CE  requires  the  operational 
evaluator  to: 

a.  Identify  events  necessary  to  verify  the  adequacy  of 
developing  system  attributes  (e.g.,  mission  performance, 
training,  RAM,  tactics,  software) . 

b.  Cause  the  early  execution  of  such  events  before  lOT. 

c.  Monitor  the  events  and  assess  the  adequacy  of  the  system 
with  respect  to  its  attributes. 

d.  Monitor  the  corrections  applied  and  assess  the  adequacy  of 
the  corrective  actions  to  be  identified  deficiencies. 

e.  Periodically  report  on  the  status  of  the  system  with 
respect  to  its  technical  and  operational  attributes. 

d.  CE  thus  requires  frequent  assessments  of  technical  and 
operational  status,  as  well  as  reports  of  that  status  to  the 
acquisition  community  and  decision  makers.  Feedback  from  these 
assessments  will  facilitate  timely  correction  of  problems  and 
help  ensure  that  a  mature  system  can  be  operationally  tested 
prior  to  the  scheduled  production  decision  milestone. 

2-11.  Frequency  of  reporting 
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CE  requires  effective  use  of  all  data  sources,  existing  Army 
resources  and  expertise,  analysis  capabilities,  modeling 
capabilities,  and  instrumentation.  CE  demands  effective  exchange 
of  knowledge,  expertise,  resources,  and  planning  throughout  the 
acquisition  community.  Thus,  it  is  imperative  that  the  evaluator 
examine  each  operational  requirement  and  its  justification; 
examine  the  causes  of  problems;  assess  the  operational  impact  of 
deficiencies;  and,  validate  the  adequacy  of  corrections  applied 
to  identified  deficiencies. 

2-12.  Objective  of  CE 

a.  The  objective  of  CE  is  to  provide  decision  makers, 
materiel  developers,  logisticians,  trainers,  combat  developers, 
and  other  acquisition  team  members  with  continuous  assessments  of 
the  system's  operational  effectiveness  and  suitability  throughout 
the  acquisition  cycle. 

b.  Based  on  multiple  data  sources,  the  system  assessments  are 
composed  of  requirements  analyses,  studies,  tactical  and 
logistical  modeling.  Early  User  Test  and  Experimentation  (EUTE) , 
contractor  tests.  Developmental  Tests  (DT) ,  Force  Development 
Test  and  Experimentation  (FDTE) ,  and  Initial  Operational  Test  and 
Evaluation  (lOTE) ,  Follow-on  Operational  Test  and  Evaluation 
(FOTE) ,  and  post-fielding  Sample  Data  Collection  (SDC) .  The 
assessments  provide  decision  makers  with  a  comprehensive 
assessment  of  a  developing  system's  ability  to  meet  the  stated 
need  in  its  current  state  and  estimates  the  potential  for  a 
successful,  mature  configuration. 

c.  The  extension  of  operational  evaluation  to  include  data 
sources  outside  the  realm  of  classic  DT  and  OT  allows  periodic 
independent  evaluation  reporting  and  facilitates  a  continuous 
interaction  between  the  MATDEV  and  the  CBTDEV.  The  operational 
evaluator  serves  as  a  catalyst  by  reporting  operational 
effectiveness  and  sni  bility  trends  which  take  into  account  the 
maturity  of  the  mate  1  system  and  its  support  concepts. 

2-13.  Principal  CE  p  ticipants. 

The  principal  CE  participants  and  their  basic  responsibilities 
are  described  as  part  of  the  T&E  community  in  Part  One  of  this 
pamphlet. 

2-14.  CE  process 

a.  To  be  effective,  operational  evaluators  must  know  how  to 
influence  the  process  and  at  what  point  and  time  they  should 
present  their  positions.  They  must  be  knowledgeable  about  the 
appropriate  source  to  use  when  obtaining  essential  information. 


2-4 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


They  must  also  be  cognizant  of  what  analyses,  studies,  tests,  and 
other  activities  are  being  planned  that  can  be  of  value. 

b.  CE  works  within  the  MAP.  AR  70-1  and  AR  73-1  detail  both 
the  traditional  MAP  and  the  Army  Streamlined  Acquisition  Process 
(ASAP) .  CE  requires  evaluators  to  assess  issues  and  criteria, 
acquisition  strategies,  test  plans,  instrumentation  plans,  threat 
definition,  software  development,  MANPRINT,  Integrated  Logistics 
Support  (ILS) ,  training  adequacy  and  many  other  program  elements 
which  can  have  a  significant  impact  on  both  the  success  of  the 
system  and  the  success  of  the  evaluations. 

2-15.  CE  for  ACAT  III  and  IV  programs 

CE  for  abbreviated  evaluate  systems.  CE  for  these  systems  is 
less  extensive,  uses  fewer  resources,  and  is  tailored  to  the 
system.  The  independent  operational  evaluator  (lOE)  is 
responsible  for  performing  CE  on  all  ACAT  III  and  IV  systems  that 
are  designated  for  DOT&E  oversight.  The  lOE  may  also  be 
requested  to  evaluate  Customer  Tests,  FDTE,  and  other  tests  in 
which  the  Program  Manager  or  MATDEV  seeks  the  evaluators 
expertise. 


Section  IV 

OTE  Planning  Process 


2-16.  Overview 

a.  This  section  details  planning  processes  used  to  develop 
the  necessary  evaluation  strategy  for  effective  evaluation  of  a 
system  and  to  derive  and  document  system  evaluation  and  test 
plans. 


b.  An  evaluation  strategy  provides  an  overview  of  the  MAP 
from  the  evaluator  perspective;  defines  the  evaluation  support  to 
be  provided  to  the  acquisition  decision  process;  and  identifies 
the  necessary  test,  model,  simulation,  and  analytic  events  needed 
to  support  the  evaluation  process.  To  develop  the  evaluation 
strategy,  the  evaluator  must: 

(1)  Review  requirements  documentation  and  COIC. 

(2)  Develop  additional  operational  issues  and  criteria 
(AOIC)  for  evaluation. 

(3)  Identify  the  data-generating  events  needed  to  answer 
the  criteria. 
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(4)  Coordinate  with  the  user  and  within  the  acquisition 
community. 

(5)  Develop  the  Operational  Test  and  Evaluation  Outline 
(Part  IV)  of  the  TEMP. 

2-17.  Principles 

a.  The  evaluation  strategy  is  developed  in  parallel  with  the 
AS  so  that  they  support  each  other.  The  evaluation  strategy 
includes  the  refinement  of  the  planned  evaluation  support  to  be 
provided  to  the  acquisition  decision  body,  and  the  refinement  of 
the  testing,  modeling,  simulation,  and  analytic  events  necessary 
to  support  the  specific  evaluation. 

b.  See  Section  VI,  below,  on  the  de  elopment  of  an  OTE 
strategy  for  Part  IV  of  the  TEMP.  The  TEMP  documents  the 
different  OTE  cycles  to  be  performed  during  the  development  and 
acquisition  of  the  system.  The  OTE  cycle  begins  with  planning  to 
develop  a  TEP  for  an  operational  test.  The  cycle  ends  either 
with  an  evaluation  or  assessment  and  the  associated  lOE  position 
and  briefing  to  the  acquisition  decision  body. 

c.  All  aspects  of  operational  effectiveness  and  suitability 
must  be  evaluated  under  anticipated  combat  conditions  or 
conditions  of  use.  Operational  evaluations  reflect  the  system  in 
a  realistic  environment  with  typical  users,  support,  and  threat 
personnel  and  equipment.  Credible  test  and  evaluation  is  highly 
dependent  on  how  well  a  realistic  operational  environment  can  be 
duplicated. 

d.  OTE  quality  are  reflected  in  the  degree  to  which  the  final 
product  conforms  to  established  scientific  standards.  Testing 
supports  good  evaluation  processes  in  providing  "ground  truth"  to 
the  summative  evaluations  that  judge  if  the  product  improves 
mission  accomplishment.  Formative  evaluation  allows  decisions  ■ 
be  structured  about  product  and  process  evaluation 
simultaneously. 


Section  V 

Development  of  a  Life  Cycle  Evaluation  Strategy 


2-18.  Coordinated  evaluation  development 

Development  and  use  of  system  requirements  and  of  issues  and 
criteria  are  integral  parts  of  the  acquisition  process.  The 
evaluator  must  both  make  use  of  and  interact  with  the  CBTDEV 
requirements  and  COIC  processes.  In  conjunction  with  COIC  and 
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requirements  developed  by  CBTDEV  and  TNGDEV,  the  evaluator 
develops  and  uses  baseline  correlation  matrixes  (BCM)  for 
tracking  requirements.  The  evaluator  ensures  that  evaluation 
issues  respond  both  to  requirements  and  to  COIC. 

2-19.  Requirements 

a.  As  the  Army  CBTDEV  for  materiel  systems,  TRADOC  develops 
the  requirements  for  new  systems  or  upgrades  to  existing  systems 
using  the  Concepts  Based  Requirements  System  (CBRS) .  Evaluation 
strategy  development  begins  during  the  requirements  development 
process  to  ensure  that  the  system  decision  milestones  are 
properly  supported  by  OTE  events  and  that  system  requirements  are 
stated  in  clear,  concise,  and,  where  appropriate,  measurable 
operational  terms. 

b.  Each  requirements  document  generated  must  be  reviewed  to 
develop  a  sound  evaluation  strategy  and  ensure  inconsistencies  in 
the  specification  of  requirements  are  resolved.  This  review  also 
determines  how  to  best  support  the  strategy  and  to  justify  any 
need  for  changes  to  milestones  or  events. 

c.  Part  One  outlines  what  to  look  for  in  requirements 
documents  and  how  to  use  the  information.  The ^primary 
requirements  documents  addressed  by  the  operational  evaluator  are 
the  Mission  Need  Statement  (MNS) ,  the  Operational  Requirements 
Document  (ORD) ,  the  System  Specifications  (Specs),  and  the 
Request  for  Proposal  (RFP) . 

d.  A  parallel  process  is  used  for  requirements  documentation 
for  IMA  systems  (see  Part  One  for  details) . 

2-20.  Critical  operational  issues  and  criteria  (COIC) 

a.  The  materiel  acquisition  decision-making  process  for 
developmental  and  NDI  systems  is  based  primarily  on  first 
analyzing,  then  evaluating,  data  associated  with  the  COIC.  The 
COIC  are  derived  from  the  operational  requirement  and  reflect  the 
minimum  essential  operational  concerns  and  standards  requiring 
answers  during  the  operational  evaluation  to  make  the  *'Beyond 
LRIP"  (MS  III)  decision. 

b.  The  approved  COIC  are  used  to  determine  the  scope, 
emphasis,  and  intensity  of  the  OTE  effort.  This  determination  is 
the  basis  for  the  resources  (personnel,  time,  facilities, 
equ<ipment,  instrumentation,  and  funds)  that  must  be  committed  to 
obtain  the  data  to  answer  the  issues  and  evaluate  the  degree  to 
which  the  criteria  are  met.  Detailed  guidance  for  preparation, 
coordination,  and  approval  of  COIC  is  provided  in  Part  Three. 
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2-21.  OTE  cycle  . 

CE  is  conducted  throughout  the  acquisition  process.  CE  is  the 
mechanism  by  which  the  independent  evaluator  follows  a  system 
throughout  its  life  cycle.  As  the  system  progresses  through  the 
acquisition  process,  the  evaluation  matures  through  plans  and 
reports.  Figure  2-1  illustrates  how  the  components  of  the 
process  follow  each  other,  focusing  on  the  TEMP,  EUTE,  lOTE,  and 
FOTE.  The  diagram  shows  the  OTE  cycle  that  occurs  for  lOTE  and 
FOTE.  This  cycle  of  plans,  events,  reviews,  and  reports  is  a 
major  component  of  the  evaluation  of  operational  effectiveness 
and  suitability  for  a  system. 


Section  VI 

Development  of  an  OTE  Strategy  (Part  IV  of  the  TEMP) 


2-22.  General 

a.  The  lOE  is  charged  with  the  development  of  Part  IV  of  the 
TEMP.  Part  Two  provides  detailed  guidance  on  the  TEMP  to  include 
Part  IV. 

b.  In  order  to  provide  evaluator  input  to  the  TEMP,  the 
evaluator  develops  the  evaluation  strategy  (to  include  pertinent 
testing)  for  the  system.  The  evaluation  strategy  is  developed 
after  MS  0  and  is  continually  updated  as  the  system  acquisition 
evolves  and  the  TEMP  is  revised. 

c.  FDTE  and  CEP  conducted  prior  to  MS  0  are  shown  as  OT&E  to 
date  in  the  TEMP,  but  are  not  a  part  of  the  evaluation  strategy. 

2-23.  Evaluation  strategies. 

a.  Full  evaluation.  Full  evaluate  =^^tems  will  require 
active  evaluator  involvement  in  all  lil  'ycle  events.  Full 
evaluate  systems  require  evaluations  ar  -sessments  to  support 
all  milestones  and  any  other  life  cycle  cisions.  The  most 
important  of  the  full  evaluate  systems  require  daily  evaluator 
involvement  and  frequent  status  reports  and  reviews.  See  Chapter 
1  for  designation  of  full  evaluate  systems. 

b.  Abbreviated  evaluation.  Abbreviated  evaluate  systems  will 
require  periodic  evaluator  involvement.  Life  cycle  events  will 
be  handled  by  document  review  and  correspondence.  Abbreviated 
assessments  are  used  to  support  milestones. 

2-24.  Testing  strategy 
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When  developing  the  OTE  strategy  for  a  system,  the  evaluator  must 
plan  for  testing  in  order  of  importance.  Rather  than  working 
sequentially  through  the  life  cycle,  the  evaluator  begins  with 
lOT  prior  to  MS  III.  EUTE  prior  to  MS  II,  FOT  after  MS  III,  and 
LUT  prior  to  LRIP  Release  are  considered  in  sequence.  The 
evaluator  can  then  consider  any  testing  prior  to  MS  I  and  any 
additional  FDTE  or  CEP  which  are  required  by  the  combat 
developer. 


2-25.  ACAT  I,  ACAT  II,  and  DOTE  oversight  ACAT  III  and  IV 
materiel  systems  (full  evaluate  systems). 

a.  These  systems  are  normally  high  cost,  high  visibility, 
technologically  advanced,  or  operationally  significant  weapons 
systems.  They  may  possess  a  combination  or  all  of  the  above 
attributes.  They  will  require  the  highest  level  of  continuous 
evaluation.  They  will  normally  have  an  extensive  series  of  tests 
throughout  the  life  cycle. 

b.  All  full  evaluate  systems  should  require  a  dedicated  lOT. 
Only  in  rare  circumstances  will  a  combined  DT/IOT  be  considered 
appropriate  for  full  evaluate  systems.  In  the  development  of  the 
evaluation  strategy,  the  evaluator  includes  an  lOT  prior  to  MS 
III  in  future  testing  plans. 

c.  Depending  on  the  complexity  of  the  system,  maturity  of  the 
technology,  and  expressed  high  level  interest,  the  evaluator 
judges  the  requirements  for  EUTE  prior  to  MS  II.  Depending  on 
the  circumstances,  this  EUTE  may  be  stand  alone,  it  may  be  a 
phase  of  DT,  it  may  be  combined  with  DT,  or  it  may  be  a  FDTE  or 
CEP  conducted  in  support  of  the  combat  developer .  The  selection 
of  the  appropriate  level  of  EUTE  is  dependent  on  the  issues  which 
must  be  addressed  prior  to  MS  II.  These  issues  need  not  be  part 
of  the  COIC,  but  may  be  other  operational  issues  or  programmatic 

issues. 

d.  For  the  most  important  systems,  FOT  is  planned  from  the 
start.  For  other  systems,  FOT  is  planned  contingent  on  system 
performance  in  EUTE  and  lOT.  FOT  need  not  be  a  full  test,  but 
may  address  only  certain  issues  not  answered  affirmatively  prior 

to  MS  III. 

e.  Systems  may  require  LUT  (stand  alone  or  in  conjunction 
with  DT)  prior  to  the  LRIP  Release  Decision.  This  testing  is 
dependent  on  the  characteristics  of  the  system  and  performance  in 
DT  and  EUTE. 

f.  Any  operational  testing  prior  to  MS  I  would  be  in  the  form 
of  EUE,  FDE,  or  CEP.  This  is  rare  and  would  be  planned  into  the 
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TEMP  only  in  exceptional  cases  to  address  issues  which  must  be 
answered  prior  to  MS  I . 

g.  The  evaluator  must  work  with  the  combat  developer  to  plan 
appropriate  FDTE  and  CEP  in  support  of  the  system.  CEP  may  be 
run  prior  to  MS  I  or  MS  II.  FDTE  is  appropriate  before  any 
milestone.  FDTE  is  particularly  important  prior  to  the  lOT  to 
assure  that  tactics  and  doctrine  are  mature  and  ready  for  play  in 
lOT. 

h.  Given  the  above  considerations,  the  evaluator  designs  the 
OT&E  strategy  and  documents  the  strategy  in  Part  IV  of  the  TEMP. 

i.  System  modifications  (see  Part  One,  chapter  7)  to  full 
evaluate  systems  may  be  handled  as  full  evaluate  OTE  or  as 
abbreviated  evaluate  OTE  depending  on  the  extent  of  the  changes 
and  the  operational  issues  engendered  by  the  changes. 

2-26.  Other  ACAT  III  and  IV  materiel  systems  (abbreviated 
evaluate  systems) . 

a.  Abbreviated  evaluate  (AE)  systems  do  not  meet  the 
requirements  for  full  evaluate  systems.  Evaluation  is  limited  to 
III  at  each  milestone.  They  will  normally  not  have  an  extensive 
series  of  tests  throughout  the  life  cycle. 

b.  Some  AE  require  a  dedicated  lOT.  In  many  cases,  a 
combined  DT/IOT  might  be  considered  appropriate.  In  certain 
instances  (NDI) ,  there  might  be  no  operational  testing  required. 
In  the  development  of  the  evaluation  strategy,  the  evaluator 
includes  appropriate  lOT  prior  to  MS  III  in  the  future  testing 
plans. 

c.  Depending  on  the  complexity  of  the  system  and  maturity  of 

the  technology,  the  evaluator  judges  the  requirements  for  EUTE 
prior  to  MS  II.  Ir  ost  cases,  AE  systems  will  have  no  EUTE. 
Depending  on  the  c  tvmstances,  this  EUTE  may  be  stand  alone,  it 
may  be  a  phase  of  it  may  be  combined  with  DT,  or  it  may  be  a 

FDTE  or  CEP  conducts i  in  support  of  the  combat  developer.  The 
selection  of  the  appropriate  level  of  EUTE  is  dependent  on  the 
issues  which  must  be  addressed  prior  to  MS  II.  These  issues  need 
not  be  part  of  the  COIC,  but  may  be  other  operational  issues  or 
programmatic  issues. 

d.  For  AE  systems,  FOT  is  planned  contingent  on  performance 
in  lOT.  FOT  need  not  be  a  full  test,  but  may  address  only 
cgj^tain  issues  not  answered  affirmatively  prior  to  MS  III. 
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e.  In  very  rare  instances,  systems  may  require  LUT  (stand 
alone  or  in  conjunction  with  DT)  prior  to  the  LRIP  Release 
Decision.  This  testing  is  dependent  on  the  characteristics  of 
the  system  and  performance  in  DT  and  EUTE. 

f.  AE  systems  will  rarely  have  any  operational  testing  prior 
to  MS  I  and  planning  for  such  would  be  an  indication  that  the 
system  should  be  full  evaluate  instead  of  AE. 

g.  The  evaluator  must  work  with  the  combat  developer  to  plan 
appropriate  FDTE  and  CEP  in  support  of  the  system.  FDTE  is 
appropriate  before  any  milestone  if  the  cost  is  justified  by  the 
nature  of  the  system. 

h.  Given  the  above  considerations,  the  evaluator  designs  the 
OT&E  strategy  and  documents  the  strategy  in  Part  IV  of  the  TEMP. 

i.  System  modifications  are  handled  as  shown  above  for  AE 
systems  depending  on  the  extent  of  the  changes  and  the 
operational  issues  engendered  by  the  changes. 

j.  System  modifications  (without  operational  issues)  and  NDI 
may  not  require  any  operational  testing  (see  Part  One,  chapters  6 
and  7) .  Market  surveys  or  investigations  and  DT  alone  may  be 
sufficient.  The  evaluator  must  document  this  fact  in  Part  IV  of 
the  TEMP.  Customer  tests  in  a  user  environment  may  be  conducted 
on  these  systems. 

2-27.  Information  Mission  Area  (IMA) 

IMA  and  other  software  intensive  systems  may  require  a  multiple 
milestone  acquisition  and  OT&E  strategy.  Guidance  on  this  type 
of  system  is  provided  in  Part  One,  chapter  3  and  in  Part  Seven. 
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Chapter  3 

Operational  Test  and  Evaulation  Planning 


Section  I 
Introduction 


3-1.  Overview 

This  chapter  details  planning  processes  used  to  develop  the 
necessary  evaluation  strategy  for  effective  evaluation  of  a 
system,  to  derive  system  evaluation  plans,  to  design  appropriate 
operational  testing,  and  to  document  the  results. 

3-2.  Principles 

a.  All  aspects  of  operational  effectiveness  and  suitability 
must  be  evaluated  under  anticipated  combat  conditions  or 
conditions  of  use.  Operational  evaluations  reflect  the  system  in 
a  realistic  environment  with  typical  users,  support,  and  threat 
personnel  and  equipment.  Credible  test  and  evaluation  is  highly 
dependent  on  how  well  a  realistic  operational  environment  can  be 
duplicated. 

b.  Test  and  evaluation  quality  are  reflected  in  the  degree  to 
which  the  final  product  conforms  to  established  scientific 
standards.  Testing  supports  good  evaluation  processes  in 
providing  "ground  truth"  to  the  summative  evaluations  that  judge 
if  the  product  improves  mission  accomplishment.  Formative 
evaluation  allows  decisions  to  be  structured  about  product  and 
process  evaluation  simultaneously. 

c.  Development  and  use  of  system  requirements  and  issues  and 
criteria  are  integral  parts  of  the  acquisition  process.  The 
evaluator  must  make  use  of  and  interact  with  the  combat  and 
training  developer's  requirements  and  COIC  processes.  In 
conjunction  with  requirements  and  COIC,  the  evaluator  develops 
and  uses  baseline  correlation  matrixes  (BCM)  for  tracking 
requirements.  The  evaluator  ensures  that  evaluation  issues 
respond  both  to  requirements  and  to  COIC. 

3-3.  Operational  test  and  evaluation  cycle 

a.  lOE  is  conducted  throughout  the  acquisition  process.  CE 
(see  chapter  2)  is  the  mechanism  by  which  the  independent 
operational  evaluator  follows  a  system  throughout  its  life  cycle. 
As  the  system  progresses  through  the  acquisition  process,  the 
evaluation  matures  through  plans  and  reports. 
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b  Figure  3-1  illustrates  how  the  components  of  the  process 
follow  each  other,  focusing  on  the  TEMP,  EUTE,  lOTE,  and  FOTE. 
The  diagram  shows  the  OTE  cycle  that  occurs  for  EUTE,  lOTE,  and 
FOTE.  This  cycle  of  plans,  events,  reviews,  and  reports  is  a 
major  component  of  the  evaluation  of  operational  effectiveness 
and  suitability  for  a  system. 

(INSERT  FIGURE  3-1) 


Section  II 

Test  and  Evaluation  Plans  (TEP) 


3-4.  General  .  •,  i.  • 

The  TEP  has  replaced  the  previously  used  independent  evaluation 
plan  (lEP)  and  test  design  plan  (TDP)  for  the  documentation  of 
test  requirements  and  planning  results.  The  abbreviated  TEP 
(ATEP)  format  used  for  abbreviated  evaluate  systems  has  been 
incorporated  within  the  TEP  and  will  no  longer  be  used. 
Implementation  of  the  TEP  process  has  changed  many  aspects  of  p® 
preparation  of  the  planning  documentation.  The  magnitude  of  the 
changes  depend  upon  the  type  and  management  level  of  the  test. 


3-5.  Level  of  detail  ^  .  4.v,=4. 

The  degree  of  detail  required  for  the  TEP  is  increased  over  that 
which  was  required  in  the  TDP.  The  evaluator  and/or  test  officer 
must  document  more  details  of  planning  developments  and 
requirements  in  the  TEP.  Consequently,  the  requirements  for  the 
DTP  have  been  reduced  to  match  the  increase  required  for  the  TEP. 


3-6.  Responsibility  for  preparation  , 

For  the  full  evaluate  level  materiel  and  IMA  systems,  the  TEP  is 
prepared  jointly  by  the  evaluator  and  tester.  For  abbreviated 
evaluate  (AE)  level  materiel  and  IMA  systems,  the  TEP  is  normally 
prepared  solely  by  the  tester.  For  FDTE,  CEP,  and  '  -her  user 
tests,  the  TEP  is  normally  prepared  solely  by  the  tsr. 


3-7.  TEP  purpose  and  content 

a  The  purpose  of  the  TEP  is  similar  for  all  conditions  and 
provides  general  information  on  the  materiel  or  IMA  system  or 
nonmateriel  requirement  description,  operational  issues  and 
criteria,  description  of  available  and  required  data  sources,  and 
a  test  design  consisting  of  a  test  concept  for  those  issues  and 
criteria  which  are  to  be  tested.  Additionally,  the  full  evaluate 
level  TEP  provides  an  evaluation  strategy  and  concept  for  the 
independent  operational  evaluation. 
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b.  Specifically,  the  TEP  shows  what  questions  will  be 
addressed  and  how  they  will  be  addressed.  For  those  questions  to 
be  addressed  through  user  testing,  the  TEP  defines  the  amount  and 
type  of  testing  to  be  conducted,  specifies  the  data  to  be 
collected,  the  factors  and  conditions  that  will  govern  test 
execution,  the  required  sample  sizes,  and  describes  how  the  test 
will  be  conducted.  The  sophistication,  complexity,  and  depth  of 
the  TEP  will  vary  greatly  as  a  function  of  the  complexity  of  the 
system. 


c.  A  TEP  provides  the  conditions,  range  of  conditions,  event 
matrixes  and  data  requirements  for  a  user  test  of  the  evaluation 
issues.  The  tester  may  determine  there  are  limitations  that 
preclude,  completely,  or  in  part,  the  address  of  a  particular 
issue  in  this  user  test.  The  evaluator  is  informed,  and  the 
limitation  is  fully  explained  in  the  TEP. 

d.  The  results  and  products  of  the  planning  activities 
discussed  below  are  the  source  material  for  the  completion  of  the 
TEPi 


3-8 .  TEP  format 

The  general  format  for  the  TEP  is  contained  in  figure  3-2.  As 
noted  above,  this  format  is  used  for  all  types  of  operational 
tests.  The  detailed  instructions  for  completion  of  each 
paragraph  of  the  TEP  format  are  contained  in  following  paragraphs 
and  figures  of  this  chapter. 

(INSERT  FIGURE  3-2) 

3-9.  TEP  paragraph  responsibilities 

The  matrix  of  responsibility  for  preparation  of  paragraphs  and 
appendixes  of  the  TEP  for  each  of  the  general  categories  of  user 
tests  is  contained  in  figure  3-3. 

(INSERT  FIGURE  3-3) 

3-10.  Purpose  of  the  TEP  (full  evaluate  systems) 

The  purpose  of  the  TEP  is  to  document  the  requirements  for  a  full 
evaluate  materiel  or  IMA  system  acquisition  program  test  and 
evaluation  (normally  for  an  EUT,  EUE,  LUT,  lOT,  or  FOT) . 

Specific  purposes  of  the  TEP  for  the  full  evaluate  level  are: 

a.  To  document  the  evaluation  and  analysis  planning  which 
supports  the  TER,  permitting  responsible  review  and  providing 
continuity  when  evaluation  and  test  personnel  change. 
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b.  To  ensure  that  the  operational  evaluator  and  the 
operational  tester  cooperate  to  efficiently  and  thoroughly  plan 
for  the  test,  analysis,  and  evaluation  to  support  the  TER. 

c.  To  present  the  test  planning  necessary  that  will  ensure 
the  test  satisfies  the  purpose  and  collects  the  data  necessary  to 
address  the  issues  and  criteria  and  support  the  intended 
evaluation. 

d.  To  provide  the  information  necessary  for  the  execution  of 
the  test. 

e.  To  provide  the  design  and  structure  of  the  TER. 

3-11.  Writing  the  TEP  (full  evaluate  systems) 

a.  Preparation  of  the  TEP  demands  close  coordination  between 
•the  operational  evaluator  and  the  operational  tester.  Chapters  1 

and  2  and  appendixes  A  through  E,  R,  and  T  through  X  of  the  full 
evaluate  system  TEP  are  the  primary  responsibility  of  the 
evaluator;  chapter  3  and  the  remaining  appendixes  are  the  primary 
responsibility  of  the  operational  tester.  Chapters  1  and  2  must 
be  completed  prior  to  completing  chapter  3,  but  is  not  a  "heel- 
to-toe"  process. 

b.  Draft  chapters  1  and  2  must  be  provided  to  the  tester 
early  in  the  process  to  guide  development  of  the  draft  chapter  3. 
Finalization  of  the  three  chapters  results  from  a  cooperative 
process  between  the  evaluator  and  tester  to  identify  and  resolve 
problems.  TEP  preparation  must  be  a  team  effort  between  the 
evaluator  and  the  tester  to  avoid  duplication  of  effort,  to 
maximize  efficiency,  and  to  minimize  problems.  By  working 
together,  the  tester  is  aware,  early  on,  of  the  evaluator’s  plan 
for  evaluation  and  test  concept.  Likewise,  by  following  the 
development  of  chapter  3,  the  evaluator  ensures  that  the  test 
planning  is  sufficient  to  sup*  "t  the  evaluation. 

c.  The  milestone  require!,  s  for  development,  processing  and 
staffing,  and  approval  of  the  EP  are  contained  in  figure  3-4. 

(INSERT  FIGURE  3-4) 

3-12.  TEP  Coordination  and  Staffing  (full  evaluate  systems) 

a.  Informal  drafts  of  chapters  and  appendixes  of  the  TEP  will 
be  coordinated  between  the  evaluator  and  tester  as  required  and 
in  accordance  with  the  TEP  milestone  schedule.  The  purpose  of 
the  coordination  is  to  provide  early  information  flow  between  the 
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tester,  evaluator,  and  other  agencies,  as  appropriate,  on  the 
plans  and  requirements  for  the  operational  evaluation  and  test. 

b.  Final  draft  TEPs  will  be  prepared  jointly  by  the  evaluator 
and  tester  within  OPTEC  or  other  OTiE  activity.  When  the 
operational  tester  and  evaluator  have  completed  an  agreed  upon 
draft,  the  TEP  will  be  simultaneously  staffed  within  the 
evaluator  and  tester  activities  for  concurrence.  Other  commands 
will  staff  in  accordance  with  command-established  procedures. 

The  draft  TEP  will  be  provided  to  TIWG  members  for  review  and 
comment . 

c.  OPTEC  distribution  lists  for  both  draft  and  approved  full 
evaluate  level  TEPs  are  contained  in  figure  3-5.  other  commands 
will  modify  these  lists  as  appropriate. 

(INSERT  FIGURE  3-5) 

d.  Comments  which  identify  problems  resulting  from  staffing 
of  the  draft  TEP  will  be  addressed  by  the  evaluator  or  tester,  as 
appropriate.  A  joint  test  and  evaluation  working  group,  chaired 
by  the  evaluator,  may  be  conducted  to  resolve  problem  areas, 
incorporate  changes,  and  ensure  the  final  draft  fully  supports 
the  evaluation  and/or  test  of  the  system. 

e.  Coordinating  drafts  will  be  staffed  in  accordance  with  and 
in  the  number  of  copies  required  in  the  staffing  list  for  draft 
TEPs  contained  in  figure  3-5.  Coordinating  copies  will  be 
distributed  by  formal  memorandum  stating  the  purpose  of  the 
staffing,  the  suspense  date  for  the  submission  of  comments,  and 
the  point  of  contact  for  questions  and  coordination. 

f.  The  contents  of  the  coordinating  drafts  should  be 
concurred  with  by  both  the  test  officer  and  independent  evaluator 
before  staffing.  Comments  returned  as  a  result  of  staffing 
should  be  incorporated  in  the  final  draft  or  the  reason  for 
nonincorporation  discussed  with  the  agency  submitting  the 
comment . 

3-13.  TEP  approval  (full  evaluate  systems) 

a.  TEPs  for  full  evaluate  level  systems  originating  within 
OPTEC  must  be  jointly  approved  by  the  Commander,  TEXCOM  and  the 
Commander,  OEC  prior  to  release  by  the  Commander,  OPTEC.  The 
tester  will  publish  and  distribute  the  approved  TEP  in  accordance 
with  the  distribution  list  for  approved  TEPs  contained  in  figure 
3-5.  The  distribution  list  is  the  minimum  required  and  should  be 
modified  to  meet  any  specific  distribution  requirements. 
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b.  After  OPTEC  TEP  approval  for  DOTE  oversight  tests,  the  TEP 
will  be  forwarded,  at  the  appropriate  time  to  the  DUSA(OR)  for 
submission  to  the  DOTE  for  approval  of  the  test  concept  and  test 
design.  DOTE  letter  of  approval  will  be  appended  to  the  approved 
TEP  as  appendix  T. 

c.  Approval  by  commands  other  than  OPTEC  will  be  accomplished 
in  accordance  with  procedures  established  by  the  command. 

3-14.  Purpose  of  the  TEP  (Abbreviated  Evaluate  Systems) 

The  purpose  of  the  TEP  for  other  than  full  evaluate  level  systems 
is  to  document  the  OT  requirements  for  an  abbreviated  evaluate 
materiel  or  IMA  system  acquisition  program.  The  specific 
purposes  of  the  TEP  are: 

a.  To  document  the  planning  which  supports  the  preparation  of 
the  TER. 

b.  To  present  the  test  planning  necessary  to  ensure  that  the 
test  will  satisfy  the  purpose  and  collect  the  data  necessary  to 
answer  the  issues  and  criteria. 

c.  To  provide  the  information  necessary  for  the  execution  of 
the  test. 

d.  To  provide  the  methodology  for  the  analysis  of  the  test 
data  and  the  preparation  of  results  and  assessments  for  the 
Qj^lteria  and  issues  and  the  overall  operational  effectiveness  and 
suitability  assessment  for  the  system. 

3-15.  Writing  the  TEP  (abbreviated  evaluate  systems) 

TEXCOM  (or  other  test  activity  as  appropriate)  normally  prepares 
the  TEP  in  its  entirety.  Direct  coordination  with  other  agencies 
gjj-g  performed  as  necessary  to  complete  the  TEP  development.  The 
TEP  wi3  nclude  the  procedures  necessary  for  the  tester  to 
determ;  onclusions  and  make  assessments  for  the  issues  and 
criteri  or  the  test,  as  appropriate.  The  milestone 

requirenu . res  for  development,  processing  and  staffing,  and 

approval  of  the  TEP  are  contained  in  figure  3-6. 

(INSERT  FIGURE  3-6) 

3-16.  TEP  coordination  and  staffing  (abbreviated  evaluate 
systems) 

a.  Within  OPTEC,  draft  TEPs  will  be  prepared  by  the  tester 
and  submitted  to  TEXCOM  for  staff  coordination  and  review. 
Evaluator  and  user  inputs  are  required  for  specific  paragraphs 
per  figure  3-3.  Coordination  with  other  agencies  will  be  in 
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accordance  with  figure  3-7.  Final  draft  TEPs  will  be  submitted 
to  Commander,  TEXCOM  for  approval. 

(INSERT  FIGURE  3-7) 

b.  Other  Army  test  organizations  will  follow  their  internal 
staffing  and  approval  procedures  for  TEPs  for  other  than  full 
evaluate  level  systems. 

c.  Distribution  lists  for  both  draft  and  approved  TEPs  are 
contained  in  figure  3-7. 

3-17.  TEP  approval  (abbreviated  evaluate  systems) 

a.  Within  OPTEC,  the  TEP  for  all  other  than  full  evaluate 
level  tests  is  approved  by  the  Commander,  TEXCOM.  The  tester 
will  publish  and  distribute  the  approved  TEP  in  accordance  with 
the  distribution  list  for  approved  TEPs  contained  in  figure  3-7. 
The  distribution  list  is  the  minimum  required  and  should  be 
modified  to  meet  any  specific  distribution  requirements. 

b.  The  approval  process  for  TEPs  for  other  than  full  evaluate 
level  systems  originated  by  other  DA  test  activities  will  follow 
internally  developed  procedures. 

3-18.  TEP  for  other  tests 

Writing,  staffing,  coordination  and  approval  of  these  TEP  will 
follow  the  same  procedures  as  those  used  for  an  abbreviated 
Evaluate  TEP.  The  purpose  of  these  TEP  is  to  document  the  test 
requirements  for  FDTE,  CEP,  CT,  and  other  appropriate  user  test 
projects.  The  specific  purposes  of  the  TEP  are: 

a.  To  document  the  planning  which  supports  the  preparation  of 
the  TR. 


b.  To  present  the  test  planning  necessary  to  ensure  that  the 
test  will  satisfy  the  purpose  and  collect  the  data  necessary  to 
answer  the  issues  and  criteria. 

c.  To  provide  the  information  necessary  for  the  execution  of 
the  test. 

d.  To  provide  the  methodology  for  the  analysis  of  the  test 
data  and  the  preparation  of  results  and  asses  sr'^nts  for  the 
criteria  and  issues. 

3-19.  Tailoring  the  TEP  for  differing  OT&E  strategies. 
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The  TEP  is  a  multipurpose  document  and  may  be  used  for  a  variety 
of  purposes  than  the  traditional  plan  to  evaluate  and  test  a 
system  prior  to  a  milestone.  These  differing  purposes  include: 

a.  Development  of  an  evaluation  plan  where  a  combined  DT/OT 
will  be  executed  under  the  auspices  of  the  development  tester. 

In  these  cases,  a  separate  and  independent  operational  evaluation 
must  be  conducted.  A  skeleton  format  for  a  TEP  for  this  purpose 
is  contained  in  figure  3-8. 

(INSERT  FIGURE  3-8) 

b.  Development  of  an  evaluation  plan  where  no  user  testing 
will  be  performed.  In  these  cases  a  separate  and  independent 
operational  assessment  of  the  program  based  on  development 
testing,  market  surveys,  modeling  or  simulation,  or  programmatics 
must  be  conducted.  A  skeleton  format  for  a  TEP  for  this  purpose 
is  contained  in  figure  3-9. 

(INSERT  FIGURE  3-9) 

c.  Development  of  an  evaluation  plan  where  multiple  user 
tests  will  be  performed  in  support  of  the  same  evaluation  and 
MADP  milestone.  In  these  cases  a  separate  test  design  roust  be 
prepared  for  each  test  to  be  performed.  A  skeleton  format  for  a 
TEP  for  this  purpose  is  contained  in  figure  3-10. 

(INSERT  FIGURE  3-10) 


Section  III 

Preparation  for  TEP  Writing 


3-20.  Research  methods 

The  performance  of  initial  research  is  essential  to  provide  t’ 
evaluator  and  test  officer  with  the  background  of  system 
developments  and  the  overall  requirements  that  may  have  to  be 
addressed  during  the  OTE  process.  The  initial  emphasis  during 
this  period  should  be  to  gain  familiarity  with  the  system  or 
concept  and  the  personnel  interested  in  the  test.  Contacts 
should  be  made  with  the  MATDEV,  CBTDEV,  TNGDEV,  FORSCOM,  and  each 
other.  Technical  libraries  may  be  used  to  assist  the  evaluator 
and  tester  in  locating  any  documents  from  previous  or  related 
tests.  Participation  in  SAG  and  TIWG  provide  additional 
opportunities  for  information  gathering. 

3-21.  TEMP 


3-8 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


The  TEMP  is  the  basic  document  for  all  test  and  evaluation 
related  to  a  system  and  is  required  for  all  materiel  acquisition 
system  programs  and  IMA  programs.  TEMP  are  not  required  for 
FDTE,  CEP,  or  CT  requirements  which  are  not  a  part  of  an 
acquisition. 

a.  TEMP  are  prepared  by  the  program  sponsor  with  major  inputs 
from  operational  and  technical  evaluators  and  testers.  TEMP  are 
approved  by  DA  and  DOD  (for  oversight  systems)  and  by  the  PEO  or 
MACOM  for  all  others.  See  Part  Two  for  a  detailed  discussion  of 
the  format,  content,  and  staffing  of  TEMP.  The  TEMP  provides 
information  in  the  following  areas: 

(1)  Relates  OTE  efforts  to  the  critical  issues  and  shows 
the  plan  to  satisfy  the  issues. 

(2)  Defines  the  T&E  schedule  and  describes  T&E  for  each 
acquisition  phase. 

(3)  Identifies  test  support  resource  deficiencies. 

(4)  Identifies  appropriate  milestone  thresholds  for  system 
requirements. 

b.  The  operational  evaluator  is  responsible  for  preparing 
Part  Four  of  the  TEMP  which  also  contains  the  CBTDEV/ functional 
proponent  COIC.  COIC  must  be  approved  prior  to  incorporation  in 
the  TEMP.  See  Part  Five,  chapter  2  for  discussion  of  evaluator 
development  of  Part  Four  of  the  TEMP. 

c.  The  TEMP  must  be  approved  prior  to  initiation  of  any 
system  testing.  No  system  test  may  be  resourced  or  initiated 
unless  that  test  is  outlined  in  an  approved  TEMP. 

d.  The  TEMP  is  coordinated  with  the  TIWG  members  and  is  the 
foundation  of  the  TEP.  As  such,  the  evaluator  and  the  tester 
should  review  the  TEMP  to  ensure  that  testing  is  not  duplicated 
and  that  agreements  on  testing  made  as  a  part  of  the  test 
integration  process  are  not  overlooked. 

3-22.  Literature  search 

The  evaluator  and  the  tester  must  be  knowledgeable  of  the  system 
or  concept  before  starting  to  develop  a  test  design.  Whenever 
possible,  they  should  review  all  documents  related  to  the  test. 
As  a  minimum,  they  should  review  the  results  of  previous  testing 
and  evaluation  and  the  requirements  documents.  These  documents 
may  be  obtained  through  technical  libraries,  the  proponent,  or 
the  program  manager. 
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a.  TER  and  TEP  of  prior  tests  may  provide  information  on  test 
and  evaluation  methodology  and  test  design.  As  a  caution, 
previous  plans  serve  only  as  a  guide  since  most  tests  and 
evaluations  require  innovative  planning,  often  with  unique^ 
methodologies.  In  some  cases,  the  results  of  earlier  testing  may 
partially  address  the  proponent's  information  needs.  This  area 
may  include  development  tests,  operational  tests,  and  tests  of 
similar  systems  or  concepts. 

b.  Requirements  documents  may  provide  information  not  found 
in  the  TEP.  These  documents  include  the  MNS,  ORD,  and,  for  older 
systems,  their  predecessor  documents.  Other  documents  that  can 
be  reviewed  are  the  COEA,  QQPRI,  and  ILSP. 

3-23.  Support  packages 

The  evaluation  planner  and  the  test  designer  must  have  all  or 
applicable  portions  of  the  support  packages  to  properly  plan  and 
design  the  test  and  prepare  a  TEP. 

a.  Support  packages  are  prepared  by  the  MATDEV,  CBTDEV,  and 
TNGDEV.  MATDEV  provide  the  system  support  package  (SSP)  and  the 
new  equipment  training  test  support  package.  CBTDEV  provide  the 
doctrinal  and  organizational  test  support  package  and  the  threat 
test  support  package.  TNGDEV  provide  the  training  support 
package.  The  TRADOC  proponent  is  responsible  for  developing, 
coordinating,  and  assembling  the  CBTDEV  and  TNGDEV  support 
packages  and  providing  these  packages  to  the  tester. 

b.  Full  evaluation  of  supportability  and  its  impact  on 
operational  suitability  will  be  accomplished  during  the  lOTE.  In 
accordance  with  AR  73-1,  all  of  the  TSP  must  be  approved  and 
provided  to  the  tester  prior  to  test  initiation.  The  OTP 
specifies  support  packages  for  test  and  suspense  dates  for  their 
submission. 

c.  A  description  and  discussion  of  r  irements  for  the 

support  packages  are  contained  in  Part  O  For  the  user|s 

convenience,  summaries  of  TSP  contents  ai  presented  in  figure  3— 

11. 


(INSERT  FIGURE  3-11) 


Section  IV 

Operational  Issues  and  Criteria  (OIC) 
3-24.  Issues  for  evaluation 
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Issues  are  questions  with  their  associated 

add^JSsed  b?  the  evaluator.  The  issues  for  evaluation  include 
both  the  COIC  developed  by  the  CBTDEV  and  the  ^ 

rriteria  fAOIC)  developed  by  the  evaluator  to  complement  and 
sSppleJ,e:“?hf?J5c  ?rc=v^r  th^  total  system  rather  than  just 
the  critical  elements.  Issues  for  evaluation  cover  all  aspects 
of  a  system  applicable  to  the  OTE  of  that  system. 

ioic‘arrdeveloped°b5  the  evaluator 

development  of  COIC  by  the  combat  developer.  will 

constructed  so  the  operational  effectiveness 

be  thoroughly  evaluated.  AOIC  are  important  to  . 

review  because  they  result  directly  in  criteria  to  be  addressed 

by  testing.  While  the  evaluator  ??WG^  the  final 

with  the  tester,  CBTDEV,  and  other  members  of  the  TIWG,  the  tinai 

pioSuc^  ii  ^hf^ole  responsibility  of  the  ^valuator  with  no 

approval  or  concurrence  outside  the  evaluator  s  command. 

i;'approved  ?Sp  identi?i2%h?L  issues  (COIC  and  AOIC)  should  be 
having  no  OT  for  the  next  milestone. 

=.i-  pKr:., 

deve?opm°nrasLssor  (TeIom)  in  coordination  with  the  development 
Of  the  COIC. 

3-28.  Operational  issues  for  FDTE,  CEP,  and  customer  tests 
Issues  for  these  tests  are  developed  by  the  test  sponsor  (CBTD  , 
TNGDEV  or  other  customer).  The  sponsor  provides  OIC  to  the 
testS.  ^hese  issues  are  not  separately  categorized  as  critical 

or  additional. 

Operational^issues^°COIC  and^AOIC)  are  questions  designed  to 
provide  information  to  a  decision  maker _ about  operational 
effectiveness  and  operational  suitability. 

a  Questions  about  the  operational  effectiveness  of  a  system 
deal  with  how  well  that  system  will  perform  what  it  is  intended 
do!  and  "hethsr  its  performance  characteristics  contribute  to 

the  force  mission. 
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b.  Questions  about  the  operational  suitability  of  a  system 
generally  address  the  significance  and  impact  of  a  system's^ 
demands  for  support  to  remain  operationally  effective.  It  is  an 
important  consideration  in  the  decision  process  to  understand, 
for  example,  what  adjustments  must  be  made  in  logistical  support 
to  service  the  new  system  without  disruption  to  service  given  to 
existing  systems. 

c.  Both  effectiveness  and  suitability  concerns  play  important 
roles  in  the  milestone  decisions.  Even  though  a  system  may  make 
significant  effectiveness  contributions  to  the  force,  the  skill 
level  required  of  the  operator  could  be  so  high  that  it  proves  to 
be  an  impractical  system  for  fielding. 

d.  Within  the  two  broad  divisions  of  effectiveness  and 
suitability  are  the  traditional  categories  for  which  every 
question  that  needs  examining  can  be  classified.  Strong 
arguments  can  be  made  that  any  suitability  category  could  also  be 
classified  as  an  effectiveness  category.  Reliability  is 
certainly  a  likely  case.  The  breakout  given  is  not  intended  to 
serve  as  a  hard  and  rigid  rule.  If  it  serves  the  purpose,  the 
evaluator  may  redistribute  the  categories. 

e.  Each  of  the  issue  categories  or  performance  attributes  are 
discussed  to  assist  in  understanding  the  meaning  of  the  category. 
The  argument  can  be  made  that  a  given  issue  is  as  appropriate 
under  any  of  several  categories  (Survivability  or  Mission 
Performance,  for  example,  can  often  be  used  to  categorize  the 
same  issue) .  A  good  rule  to  remember  is  that  the  main  purpose 
for  categorizing  issues  is  to  give  some  degree  of  order  to  the 
issues.  It  simply  makes  them  easier  to  manage  throughout  the 
test  planning,  execution,  and  reporting  process.  The  right 
question  must  be  asked,  but  not  necessarily  be  put  in  the  right 
category. 

3-30.  Mission  performance  issues 

Mission  performan>'  i',iues  are  those  that  deal  with  determining 
how  well  the  systf  ijes  what  it  is  designed  to  do.  Such  issues 
normally  address  t  major  functions  of  the  system  (e.g., 
detecting,  identifying,  and  engaging  aircraft  or  receiving, 
processing,  and  relaying  message  traffic) .  They  generally 
address  system  level  functions  and  do  not  address  component 
functions. 

3-31.  Survivability  and  vulnerability  issues 

Survivability  and  vulnerability  issues  are  those  that  deal  with  a 
system's  likelihood  of  avoiding  being  rendered  ineffective  by 
enemy  action  while  a  system  is  performing  its  mission.  The  key 
is  "likelihood"  since  no  system  has  foolproof  safeguards. 
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Operational  test  measures  are  normally  expressed  in  of  signatures 
and  exposure  times.  These  measures  determine  ease  of  enemy 
engagement.  Other  operational  measures  determine  the  extent  of 
damage,  since  the  enemy  engaged  the  system  and  because  the  enemy 
hit  the  system.  See  Part  One  for  a  more  detailed  discussion  of 
Survivability  and  vulnerability. 

3-32.  Reliability,  availability,  and  maintainability  (RAM) 
issues 

A  RAM  issue  has  three  components  -  reliability,  availability,  and 
maintainability.  These  may  be  broken  out  into  separate  issues. 
See  Part  One  for  a  more  detailed  discussion  of  RAM. 

a.  Reliability  deals  with  the  assurance  that  a  system  will 
not  encounter  an  unacceptable  number  of  failures  during  operation 
(frequency  of  failure) ,  and  is  generally  expressed  in  terms  of 
"Mean  Time  between  Operational  Mission  Failure  (MTBOMF)." 

b.  Availability  addresses  the  probability  that  a  system  can 
operate  whenever  required.  It  is  a  measure  of  usable  operating 
time  expressed  by  percent  of  total  time  (availability  * 
uptime /uptime  plus  downtime) . 

c.  Maintainability  deals  with  the  ease  of  repairing  or 
replacing  a  system  component  that  failed,  the  capability  for 
maintenance  at  the  appropriate  level,  the  capability  for 
personnel  at  these  levels  to  perform  diagnostics  or  maintenance, 
and  the  inherent  system  capability  to  be  maintained.  It  accounts 
for  the  time  required  to  diagnose  (fault  isolate,  detect,  and 
locate),  remove,  repair,  replace,  and  test  for  adequacy  of 
correction.  Maintainability  measures  are  generally  expressed  in 
Mean  Time  to  Repair  (MTTR)  or  restore  the  system  to  an  operating 
condition. 

3-33.  Logistics  supportability  issues 

Logistics  supportability  issues  deal  with  the  impact  of  providing 
maintenance  and  operating  support,  in  both  concepts  and  materiel. 
See  Part  One  for  a  more  detailed  discussion  of  ILS  and  logistics 
supportability . 

a.  Maintenance  support  includes  repair  teams,  procedures,  the 
spare  parts  supply  system,  and  materiel  evacuation  assets. 

b.  Operating  support  must  consider  such  expendable  items  as 
POL,  air  filters,  rations  and  ammunition. 

c.  Transportability  and  deployability  evaluation  issues 
address  the  ability  to  move  the  system  into  a  theater  of 
operations  and  move  it  within  the  theater  of  operations 
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consistent  with  the  mission  (these  issues  are  sometimes 
considered  as  a  separate,  distinct  element  of  operational 
suitability,  rather  than  as  a  part  of  logistics  supportability) . 
These  issues  may  deal  with  airplane  loading  or  internal  and 
external  helicopter  loads.  The  examination  must  not  only  address 
the  ability  of  aircraft  to  carry  the  load,  but  also  their 
availability  (numbers  of  carrier  vehicles  not  otherwise 
committed) . 

3-34.  MANPRINT  considerations  in  operational  issues 
MANPRINT  is  a  concept  used  to  address  human  performance 
considerations  as  they  apply  to  a  system.  MANPRINT  in  itself  is 
not  an  issue,  but  there  are  six  areas  of  interest  that  fall 
within  the  concept  and  are  considered  in  developing  operational 
evaluation  issues.  The  six  areas  of  interest  are  Manpower, 
Personnel,  Training,  Human  Factors  Engineering,  System  Safety, 
and  Health  Hazards.  MANPRINT  examines  management  and  technical 
efforts  to  assure  total  system  effectiveness  by  posing  the 
question;  "Can  typical  soldiers,  with  the  training  given,  perform 
these  tasks  to  these  standards  under  these  conditions  using  this 
equipment?”  See  Part  One  for  a  more  detailed  discussion  of 
MANPRINT. 

a.  Manpower  deals  with  the  number  of  people  in  the  force 
structure,  irrespective  of  skill  level,  required  to  sustain 
operations  under  combat  conditions,  maintain,  and  support  a 
system.  As  such,  it  seldom  has  direct  connection  with  the 
operational  evaluation  issues  for  a  system. 

b.  Personnel  addresses  the  ability  to  provide  qualified 
people  for  specific  skills  needed  to  operate,  maintain,  and 
support  a  system. 

c.  Training  considers  time  and  resources  required  to  develop 
the  correct  skill  levels. 

d.  Human  Factors  Engineering  considers  the  characteristics  of 
people  (physical,  cultural,  mental)  that  must  be  addressed  in 
designing  a  system  (known  as  an  ergonomic  science,  this  addresses 
all  aspects  of  the  soldier-machine  interface) . 

e.  System  Safety  considers  the  safety  engineering  principles 
and  standards  necessary  to  optimize  safety  within  the  bounds  of 
operational  effectiveness,  time  and  cost. 

f.  Health  Hazards  consider  conditions  that  can  cause  illness, 
disability,  or  reduced  job  performance. 

3-35.  Means  of  employment  issues 
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Means  of  employment  consists  of  organization,  doctrine,  and 
tactics.  Organization  evaluation  issues  deal  with  the  manner  of 
the  distribution  of  people  by  position  and  of  equipment  to 
optimize  the  system's  effectiveness  in  the  context  of  its 
operating  environment.  Such  issues  also  examine  the  organization 
of  the  maintenance  and  other  support  units  that  must  interact 
with  the  system's  unit.  Doctrine  issues  investigate  the  adequacy 
of  planned  doctrine  for  the  employment  of  the  system.  These 
issues  must  consider  doctrinal  aspects  of  the  unit  or 
organization  that  hosts  the  system,  as  well  as  those  aspects  of 
supporting  and  supported  units  to  optimize  the  effectiveness. 

3-36.  Interoperability  issues 

Interoperability  issues  examine  the  extent  to  which  a  system 
interacts  with  or  does  not  interfere  with  other  systems  on  the 
battlefield.  The  system  is  studied  for  its  synergistic 
relationship  in  its  operational  environment.  See  Part  One  for  a 
discussion  of  interoperability. 

3-37.  Software  considerations  in  operational  issues 
Software  considerations  for  battlefield  automated  systems,  except 
for  organization,  doctrine  and  transportability  and  deployability 
categories,  must  be  made  when  forming  the  issues.  Although 
primarily  found  in  mission  performance  functions,  software 
extends  to  the  remaining  categories.  Survivability  and 
vulnerability  issues  for  example,  may  have  a  radar  warning 
feature  supported  by  software  that  warrants  examination.  Test 
Measurement  and  Diagnostic  Equipment  (TMDE)  is  likely  to  be 
heavily  software  dependent.  Each  category  should  be  examined  to 
see  if  there  is  reason  to  include  a  software  issue  and  criteria. 
Most  software  evaluations  require  some  verification  of  the 
software's  value  through  testing.  See  Part  Seven  for  a  detailed 
discussion  of  software  in  the  T&E  process. 


Section  V 

Baseline  Correlation  Matrixes  (BCM) 


3-38.  Need  for  a  BCM  .  a.  •  i 

For  a  structured  approach  to  reviewing  and  comparing  operational 
requirements,  the  evaluator  creates  a  BCM.  The  evaluator  should 
not  wait  until  the  first  test  is  being  planned  to  develop  the 
BCM.  The  initial  BCM  should  be  developed  after  receipt  of  the 
MNS.  The  evaluator  cannot  effectively  write  AOIC  until  he  has 
developed  the  BCM.  Use  of  the  BCM  assures  that  an  evaluator's 
review  of  the  primary  operational  requirement  documents  will 
uncover  inconsistencies  to  be  brought  to  the  attention  of  the 
CBTDEV  and  the  MATDEV  for  resolution.  Using  the  BCM  throughout 
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the  CE  process  makes  it  possible  to  eliminate  ambiguity  and 
clearly  define  adequate  MOEs  that  measure  the  system  against  the 
proper  requirements. 

3-39.  Development  of  the  BCM 

For  some  systems  the  BCM  is  as  simple  as  a  matrix  with  a  column 
for  each  requirements  document  and  a  row  (or  series  of  rows)  for 
each  category  of  operational  effectiveness  and  suitability.  For 
more  complex  s,ystems,  as  complex  as  a  loose-leaf  binder  organized 
with  a  section  for  each  category  or  an  automated  data  base.  A 
BCM  is  an  organized  presentation  of  the  operational  requirements 
in  all  the  applicable  requirements  documents  and  COIC.  In 
general,  the  BCM  includes  the  MNS,  ORD,  and  COIC.  The  BCM  does 
not  normally  include  the  system  specifications  or  RFP  unless  they 
contain  operational  requirements.  For  NDI,  the  RFP  and  system 
specifications  may  be  the  primary  requirements  documents 
available  to  the  evaluator. 

3-40.  Sample  BCM 

A  sample  BCM  is  provided  in  figure  3-12.  The  operational 
requirements  and  COIC  are  indexed  to  the  individual  evaluation 
issues  (far  left-hand  column)  and  are  traced  through  the  process 
to  the  MOEs  (far  right-hand  column)  that  will  be  gathered  in 
testing.  MOEs  are  used  to  ensure  that  data  collected  are 
comprehensive  enough  to  address  all  the  different  ways  in  which  a 
requirement  may  have  been  stated. 

(INSERT  FIGURE  3-12) 

3-41.  Evolution  of  the  BCM 

Developing  a  BCM  is  an  evolutionary  process.  For  each  different 
requirements  document  and  the  COIC,  every  operational  requirement 
is  recorded  in  its  appropriate  category  with  all  available  and 
appropriate  justification.  As  the  requirements  of  each  new 
document  are  added,  they  are  compared  to  the  other  .luirements 
in  the  matrix.  By  tracing  the  consistency  of  the  irements 

for  wording,  measures,  units,  and  specific  values,  screpancies 
are  found  at  a  time  when  their  impact  can  easily  be  ,  inimized. 

If  an  inconsistency,  omission,  or  other  change  that  is  not 
directly  traceable  to  an  earlier  requirement  is  noted,  it  must  be 
justified  or  the  inconsistency  noted. 

3-42.  BCM  procedures 

The  evaluation  issues  and  MOE  are  examined  to  assure  that  each 
and  every  requirement  is  covered  by  a  critical  or  additional 
issue  and  by  a  MOE.  The  end  product  is  a  consistent,  fully 
justified  set  of  operational  requirements  that  is  a  firm 
foundation  for  an  evaluation.  A  discussion  of  BCM  procedures  is 
provided  in  figure  3-13. 
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(INSERT  FIGURE  3-13) 


Section  VI 
Development  of  AOIC 


3-43.  AOIC  subject  areas  ,  . 

Two  types  of  issues  (COIC  and  AOIC)  are  used  in  OTE  planning. 

The  evaluator  (OPTEC  or  other  OTE  command)  develops  the  AOIC 
using  methodologies  similar  to  those  used  for  COIC.  (See  Part 
Three.)  The  evaluator  coordinates  the  AOIC  with  TIWG  members  as 
a  part  of  the  TEP  (TIWG  members  comment  on,  but  cannot  concur  or 
nonconcur  with  the  TEP) .  The  following  components  are  addressed 
for  each  system  if  applicable; 

a.  Mission  performance. 

b.  Survivability  and  vulnerability. 

c.  RAM. 

d.  Logistics  supportability . 

e.  MANPRINT  (Manpower,  Personnel,  Training,  Human  Factors 
Engineering,  System  Safety,  and  Health  Hazards) . 

f .  Means  of  employment  (organization,  doctrine,  tactics) . 

g.  Software. 

h.  Interoperability. 

3-44.  Elements  of  an  AOIC  set 

The  elements  of  an  AOIC  set  are  the  issue  statement,  scope, 
criterion  or  set  of  criteria  associated  with  the  issue,  and 
rationale  for  each  criterion  (with  a  source  to  show  the  origin  of 
the  criterion) .  The  conditions  for  examining  and  standards  for 
measuring  a  comprehensive  issue  are  contained  in  the  scope  and 
cjfiteria,  respectively.  Each  element  contributes  to  the 
cohesiveness  of  a  complete  operational  evaluation  issue.  It  is 
re-emphasized  that  answers  to  an  issue  may  be  provided  by  one  or 
more  means.  Therefore,  this  paragraph  discusses  the  construction 
of  an  issue  in  its  entirety,  irrespective  of  the  intended  data 
source . 

3-45.  Issue  statements  _ 

An  issue  should  be  stated  in  the  form  of  a  question  for  which  a 
response  will  make  a  significant  contribution  to  the  decision 
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making  process.  The  issue  should  be  clear,  concise,  meaningful 
for  the  system  in  question,  thought  provoking,  and  broad  in 
nature  (the  criteria  can  narrow  the  focus  to  a  specific  path  or 
measure) . 

a.  A  well-constructed  issue  should  ask  questions  such  as 
"if,"  "whether  or  not,"  or  "how  well"  a  system  performs  its  major 
functions,  "how  likely"  it  can  survive,  "how  easily"  it  can  be 
repaired,  "whether  or  not"  the  logistical  support  concept  can 
service  its  ammunition  requirements,  "how  adequate"  the  training 
program  may  be,  or  "what  extent"  performance  is  degraded  in  an  EW 
environment.  There  are  two  basic  forms  a  question  can  take.  One 
is  "if"  or  "whether  or  not."  The  other  is  "how  well,"  "how 
easily,"  or  "to  what  extent." 

b.  Generally,  the  "if/whether  or  not"  form  is  more  suitable 
for  a  system  that  is  replacing  an  existing  system.  It  either 
performs  or  it  does  not  perform  a  function.  The  question  is; 
given  that  it  performs,  is  it  an  improvement? 

c.  "How  well/how  easily/to  what  extent"  forms  of  questions 
are  normally  more  suited  to  systems  for  which  there  is  no 
baseline  and  the  realistic  level  of  expectation  cannot  be  as 
easily  determined. 

d.  The  questions  should  be  designed  to  encourage  freedom  in 
exploring  paths  of  interest  that  will  provide  useful  operational 
information  to  the  decision  maker.  Questions  that  specify  a 
certain  condition  (reserved  for  the  criteria)  often  inhibit 
opportunities  to  discover  the  useful  outcomes.  The  consequence 
of  too  narrow  an  investigation  at  the  outset  can  lead  to  a 
partial,  incomplete,  or  even  an  irrelevant  answer. 

e.  The  question,  "Does  the  machine  gun  mount  have  to  be 
removed  prior  to  loading  the  ’ehicle  on  a  Cl 30?"  is  too 
restrictive.  Unless  there  ’  an  abundance  of  complementing 
issues,  this  issue  will  pro  i/  limit  the  investigation  to  only 
the  aspect  of  on-loading  anc  £f-loading  clearance.  The  issue 
does  not  allow  for  other  factors  that  may  need  to  be  considered 
in  the  airlift  deployability  area.  Further,  the  issue 
presupposes  that  to  remove  the  machine  gun  mount  is  a  burden. 

f.  A  better  question  is,  "Can  the  vehicle  be  strategically 
and  tactically  airlifted?"  Then,  as  shown  in  discussing  the 
scope  and  criteria,  the  essential  conditions  and  standards  can  be 
established  and  measured  respectively.  What  the  decision  maker 
needs  to  know  is  whether  the  vehicle  can  be  efficiently  airlifted 
to  and  within  the  theater  of  operations  consistent  with 
employment. 
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g.  The  number  of  issues  should  be  kept  to  a  manageable  level, 
one  must  recognize  that  if  a  question  is  important,  every 
opportunity  must  be  made  to  ensure  that  it  is  addressed.  The 
challenge  is  to  determine  what  is  important,  and  to  consider 
adequately  everything  without  diluting  the  process.  The 
judicious  selection  of  questions  (issues)  contributes  to  meet 
that  challenge. 

3-46.  Scope  statements  ^  ^ 

The  scope  enhances  the  issue  by  defining  the  context  in  which  the 
issue  is  to  be  addressed.  It  provides  guidance  concerning  the 
conditions  and  range  of  conditions  necessary  for  examining  the 
issue. 

a.  As  a  minimum,  the  scope  must  describe  the  tasks  to  be 
performed  and  the  operational  environment  in  which 
measured.  A  thorough  understanding  of  the  mission  and 
employment  is  essential  to  assure  that  the  correct  circumstances 
are  established.  The  scope  should  not  specify  how  the 
information  will  be  generated,  nor  what  measures  shall  be  made. 
(The  means  of  generating  information  are  contained  in  the  TE  , 
and  the  measures  used  are  reserved  for  the  criterion  paragraph.) 

b.  The  challenge  in  preparing  a  scope  is  to  enhance  the 
meaning  of  the  issue  such  that  it  provides  a  comprehensive 
overview  of  the  necessary  conditions  and  serves  as  a  basis  for 
resourcing  and  sizing  any  later  source  of 

simulations,  models,  studies,  operational  testing,  and  other 
testing) . 

3-47.  Criteria  statements 

The  criterion  associated  with  an  operational  issue  at  a  given 
stage  in  the  system's  development  is  an  expression  of  the  level 
of  performance  established  to  measure  the  accomplishment  or  the 
achievement  of  the  issue.  Although  it  reflects  an  expectation, 
it  should  never  be  viewed  as  a  pass/fail.  Instead,  it  provides  a 
basis  for  comparison  with  actual  performance  in  determining  t  e 
degree  of  acceptability. 

a.  A  criterion  is  either  stated  for  a  system's  functions  (the 
abilitv  to  detect,  identify  and  locate  a  hostile  aircraft)  or  by 
a  system's  characteristics  (the  ability  to  operate  for  a  given 
period  without  a  mission  failure) . 

b  There  are  two  types  of  criteria  used  to  support  answering 
an  issue,  quantitative  and  qualitative.  The  quantitative 
criterion  is  preferred  since  it  lends  itself  to  an  analysis  and 
more  credible  description  of  the  actual  outcome  of  the 
investigation,  be  it  testing,  mo^*>-  .ing  or  other.  The  qualitative 
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criterion  is  nonetheless  an  acceptable  approach  for  circumstances 
that  do  not  lend  themselves  to  assigning  numerical  value  to  the 
measure  (e.g.,  "there  shall  be  no  increase  required  of  personnel 
or  support  vehicles  to  maintain  the  expanded  communication 
equipment/power  supply") . 

c.  There  is  a  tendency  to  use  the  term  "investigative"  for 
criteria  when  the  user  wants  to  explore  the  potential  of  a 
system's  performance  or  characteristic.  Such  a  practice  has  a 
useful  application  in  FDTE  and  in  early  user  evaluations  to 
determine  the  parameter's  level  of  performance,  but  it  is  not  a 
criterion.  "Investigative"  means  that  the  user  has  no 
requirement  or  expectation  for  the  measure  but  would  like  data 
upon  which  to  determine  a  reasonable  expectation. 

d.  A  complete  criterion  statement  should  always  have  two 
elements;  a  measure  (parameter)  and  a  value  (threshold) . 
Investigative  criteria  have  no  value. 

e.  A  Measure  of  Effectiveness  (MOE)  is  normally  the  method 
for  quantifying  the  criteria.  MOEs  can  be  directly  observed, 
calculated  from  Measures  of  Performance  (MOPs) ,  or  derived  from 
military  judgment.  MOPs  contribute  directly  to  MOEs  and  provide 
the  components  for  MOE  calculation.  Sometimes,  a  third  element, 
confidence  level,  is  appropriate.  The  measure  is  the  parameter 
to  be  examined;  for  example  probability  of  kill,  time  to  repair, 
mean  time  to  process  a  request,  message  completion  rate,  percent 
of  targets  correctly  detected,  identified  and  located,  set 

up/ tear  down  times,  degree  of  degradation  (percent  of  additional 
unusable  messages)  in  an  EW  environment. 

f.  The  value  is  the  quantifiable  (numerical)  expression 
associated  with  the  measure  that  is  desired  or  acceptable  to  the 
user.  (If  the  measure  is  expected  to  improve  as  development 
continues,  then  the  value  at  any  given  point  in  time  is 

consid  ad  a  threshold  which  represents  an  intermediate 
crite:  n . ) 

g.  .  value  may  be  determined  by  comparing  the  parameter  to 
the  system  being  replaced,  e.g.,  "The  mean  time  to  repair  must  be 
no  more  than  that  of  the  ..."  or  "The  P(h)  of  the  weapon  must  be 
at  least  as  good  as  that  of  the  ..." 

h.  A  value  also  may  be  established  as  an  absolute  value  with 
no  relationship  to  anything  else.  Such  is  the  case  when  the 
parameter  to  be  measured  has  no  corresponding  parameter  from  an 
existing  system. 
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i.  Remember  that  when  the  intent  is  to  replace' the  existing 
system  with  a  new  one,  there  are  tradeoffs  to  consider.  It  may 
be  unrealistic  to  expect  every  parameter  on  the  new  system  to 
outperform  the  existing  system.  Often  there  are  many  features 
that  can  be  left  unaltered  to  achieve  one  or  two  major 
improvements.  For  example,  "The  effective  range  of  engagement 
shall  be  at  least  500  meters  greater  than  system  'y'  without 
degradation  to  Ph  or  reaction  time." 

j.  It  is  also  conceivable  that  the  new  system  needs  to  show 
no  operational  improvement  over  the  existing  system  because  the 
current  system  was  good  enough  but  the  new  system  is 
significantly  cheaper.  The  confidence  level  defines  the  degree 
of  risk  or  certainty  associated  with  the  results. 

3-48.  Rationale  statements 

Pqj-  every  evaluation  issue,  there  should  be  a  rationale  statement 
that  explains  the  reason  for  the  choice  of  the  MOE  or  MOP  and 
criterion  values.  The  rationale  serves  to  force  the  user 
(developer  of  issues  and  criteria)  to  consider  and  justify  why 
the  issue  and  criterion  is  required  or  acceptable.  Without  any 
supporting  rationale,  there  may  be  a  tendency  to  develop 
unrealistic  or  unnecessary  standards.  Careful  thought  must  be 
applied  and  explained  to  ensure  that  the  measures  and  their 
values  are  appropriate,  realistic,  and  practical. 


Section  VII 

Documentation  of  the  TEP  introduction  (chapter  1  of  the  TEP) 


3-49.  Preparation  of  chapter  1  of  the  TEP 

After  completion  of  the  above  steps,  the  evaluator  documents  the 
introduction  to  the  plan  in  chapter  1  of  the  TEP.  This  chapter 
of  the  TEP  provides  necessary  information  to  understand  the  bases 
of  both  the  test  and  of  the  evaluation.  This  chapter  may  be 
prepared  by  either  the  evaluator  or  the  tester  depending  on  the 
type  of  test  or  evaluation.  See  figure  3-3  for  allocation  of 
responsibilities  for  the  preparation  of  this  chapter.  See  figure 
3-14  for  detailed  instructions  on  the  writing  of  this  chapter. 

3-50.  TEP  purpose 

State  the  purpose  for  conducting  the  evaluation  and  for  planning 
the  testing.  State  MADP  milestones  or  other  decision  supported 
by  the  evaluation  and/or  testing  and  the  form  and  extent  of  the 
evaluation  or  assessment.  State  the  test(s)  identified  in  the 
system  TEMP  or  other  planning  document  to  be  designed  in  the  TEP. 

3-51.  TEP  scope 
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Address  the  breadth  of  the  data  sources  to  be  used  to  prepare  the 
test  and  evaluation/assessment  reports.  Provide  an  overview  of 
relevant  models,  analyses,  tests,  equipment,  and  other  resources 
which  together  are  the  basis  for  any  planned  evaluation  or 
assessment.  Address  the  extent  to  which  effectiveness  and 
suitability  can  and  will  be  evaluated. 

a.  TEP  for  Milestone  II  typically  support  evaluations  of 
specific  system  capabilities  and  potential  for  maturation.  TEP 
for  Milestone  III  typically  support  evaluations  of  operational 
effectiveness  and  suitability  and  the  full-scale  production 
decision. 

b.  When  required,  TEP  for  a  post-milestone  III  decision  is 
produced.  CBTDEV  and  TNGDEV  may  also  plan  evaluations  or 
assessments  in  support  of  their  products. 

3-52.  TEP  background 

Includes  the  background  of  the  system  development  and  the  test 
and  evaluation  of  the  system.  A  proper  review  of  the  TEMP 
(paragraph  3-21)  should  provide  adequate  information  to  complete 
this  paragraph. 

a.  Program  background.  Include  an  overview  of  the  program, 
its  acquisition  strategy,  the  system's  anticipated  use  to  the  ^ 
Army,  and  the  deficiency  identified  in  the  Mission  Area  Analysis 
(MAA)  that  the  system  is  to  correct.  Identify  the  next  program 
decision  to  be  supported  by  the  testing  and  evaluation  and  the 
decision  to  be  made  (i.e.,  enter  next  acquisition  phase,  low  rate 
initial  production,  full-scale  production,  fielding) .  Identify 
deficiencies  or  suitability  and  effectiveness  problems  existing 
in  similar  systems,  as  well  as  the  measures  used  to  evaluate 
those  systems,  if  applicable.  For  T&E  of  CBTDEV  or  TNGDEV 
products,  the  background  will  cover  the  background  of  the 
development  of  that  product. 

b.  T&E  background.  Include  a  summary  of  all  OTE,  DTE, 
contractor  testing  to  date.  Include  both  the  scope  and  the 
results  of  the  T&E. 

c.  COEA/OTE  relationship.  Required  for  any  system  for  which 
a  COEA  is  done  and  OTE  is  conducted.  Describe  the  linkage 
between  the  COEA  and  the  planned  results  of  OTE.  The  description 
of  the  linkage  should  explain  how  the  MOE  and  MOP  used  for  OTE 
are  consistent  with  the  criteria  in  the  COEA,  which  in  turn 
should  have  MOE  and  MOP  consistent  with  the  ORD,  the  TEMP  and  the 
Acquisition  Program  Baseline  (APB) . 

3-53.  System  description 
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Describe  the  system  (or  CBTDEV/TNGDEV  product) .  Describe  the 
major  roles,  missions,  and  components  or  characteristics  of  the 
system.  Describe  similarities  and  differences  between  the  sys  em 
under  test  and  the  objective  system  being  developed.  Summarize 
the  concept  for  force  structure  and  employment.  A  proper  review 
of  the  TEMP  (paragraph  3-21)  should  provide  adequate  information 
to  complete  this  paragraph. 

3-54.  Projected  threat  ^ 

Define  the  approved  threat  in  the  post-IOC  timeframe  of  the 
tested  system.  Include  capabilities,  typical  means  of  operation, 
and  known  methods  of  defeating  the  system.  For  ACAT  I  and  ACAT 
II  systems,  base  the  threat  on  the  DA  DCSINT  approved  and  DIA 
validated  threat.  For  Milestone  II,  state  potential  targets, 
countermeasures,  and  opposing  weapons  that  a  single  system  can 
expect  to  encounter  on  the  battlefield.  For  Milestone  III  and 
beyond,  describe  the  threat  to  the  system  at  battalion  level  or 
equivalent.  For  CBTDEV/TNGDEV  products  being  tested,  an 
appropriate  threat  statement  should  be  developed.  See  Part  One 
for  a  detailed  discussion  of  Threat  in  T&E. 

3-55.  T&E  milestones 

List  all  milestones  important  to  the  success  or  completion  of  the 
T&E  leading  to  a  MDR  or  other  event  supported  by  the  T&E  effort. 
Include  a  comprehensive  list  of  all  events  (study  and  analysis 
milestones,  model  completion  dates,  document  approval  milestones, 
and  test  events)  critical  to  the  overall  T&E  effort.  A  proper 
review  of  the  TEMP  (paragraph  3-21)  should  provide  adequate 
information  to  complete  this  paragraph. 

^  IMP  ^  ^  Q  J  ^ 

List  the  OIC  used  to  evaluate  the  system,  construct  an  evaluation 
plan,  and  develop  a  test  design  for  the  system  (or  CBTDEV/TNGDEV 
product) .  For  materiel  and  IMA  systems,  OIC  will  be  broken  down 
into  COIC  (prepared  by  the  CBTDEV/ functional  proponent  lAW  Part 
Three)  and  AOIC  prepared  by  the  evaluator  (lAW  Section  VI) .  For 
T&E  of  CBTDEV  or  TNGDEV  products,  OIC  developed  by  the  CBTDEV  or 
TNGDEV  are  required.  COIC  and  AOIC  collectively  constitute  the 
OIC  for  the  TEP. 

3**57  COIC 

List* the  approved  COIC  directly  extracted  from  the  approved  TEMP. 
The  COI  address  key  operational  questions  about  a  system  which 
must  be  addressed  at  each  milestone  decision.  The  COI  emphasize 
determination  of  attainment  of  certain  key  performance  levels  and 
surfacing  potential  problems  which  could  jeopardize  acquisition. 
Each  COI  must  have  at  least  one  associated  criterion.  The 
evaluator  may  add  additional  evaluator  criteria  to  a  critical 
issue.  Criteria  which  are  a  part  of  the  original  COIC  set  will 
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always  have  both  a  measure  (parameter)  and  a  threshold  (value) . 
Criteria  developed  by  the  evaluator  may  have  only  a  measure. 

3-58.  AOIC 

List  evaluator  developed  AOIC  which  complement  and  supplement 
approved  COIC  completely  addressing  all  aspects  of  operational 
effectiveness  and  suitability  of  the  system.  AOIC  address  all 
operational  questions  about  a  system  which  must  be  addressed  for 
a  complete  operational  evaluation  or  assessment  of  the  system. 

AOI  emphasize  determination  of  the  attainment  of  all  performance 
levels  and  surfacing  potential  problems  which  could  affect  the 
acquisition.  The  AOIC  may  change  from  milestone  to  milestone  as 
the  system  evolves  and  the  system  requirements  are  defined.  AOIC 
may  or  may  not  have  quantitative  criteria.  Keep  the  number  of 
issues  to  a  manageable  level.  Nonetheless,  it  is  necessary  to 
recognize  that  if  a  question  is  important,  every  opportunity  must 
be  made  to  ensure  that  it  is  addressed.  The  challenge  is  to 
determine  what  is  important  and  to  consider  adequately  everything 
without  diluting  the  process.  Evaluator  developed  criteria  may 
have  only  a  measure. 

3-59.  BCM 

Call  out  the  BCM  in  this  paragraph  and  include  the  actual  BCM  as 
a  figure  or  table  on  a  facing  page.  Only  full  evaluation  TEP 
require  a  BCM.  Section  V  contains  instructions  on  preparation  of 
the  BCM.  The  purpose  of  including  the  BCM  in  the  TEP  is  to  show 
the  derivation  of  the  AOIC  from  the  other  requirements  for  the 
system  to  include  the  MNS,  ORD,  FD,  system  specification,  RFP, 
COIC,  and  SOTEP.  The  BCM  provides  a  mapping  of  the  various 
documented  system  requirements  documents  with  the  COIC  and  the 
AOIC.  The  evaluator  uses  this  structured  process  to  uncover  and 
resolve  inconsistencies  between  the  primary  requirements 
documents  and  the  total  set  of  operational  issues  and  criteria. 

(INSERT  FIGURE  3^4) 


Section  VIII 

OTE  Concept  Development 


3-60.  General 

The  sophistication,  complexity,  and  depth  of  the  evaluation  will 
vary  as  a  function  of  the  complexity  of  the  system.  This  section 
provides  guidance  about  methods  for  development  of  the  evaluation 
concept  written  from  the  perspective  of  more  complicated  system 
evaluations  to  be  as  complete  and  all-encompassing  as  possible. 
Significant  latitude  exists  for  evaluators  to  reduce  scope  and 
complexity  of  OTE  concepts.  For  example,  the  material  on  the 
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approach  to  system  evaluation  will  likely  be  complex  for  a  major 
air  defense  missile  system.  For  an  IPR  system  that  is  not 
complicated,  however,  it  may  only  consist  of  a  statement  that 
system  effectiveness  will  be  determined  using  military  judgment 
based  on  the  soldier's  ability  to  complete  task  X.  When  the 
tester  prepares  the  analytic  concept  for  a  TEP  of  an  abbreviated 
evaluate  system  or  a  TEP  for  FDTE,  CEP,  or  CT,  he  uses  a  Pattern 
of  Analysis  (POA)  to  develop  measures  and  data  requirements. 

(See  section  XVI,  below,  for  POA  guidance.) 


3-61.  System  evaluation  concept  ,  4.  • 

The  TEP  basis  for  evaluation  planning  is  the  system  evaluation 
concept,  the  methodology  that  will  consolidate  the  results  of  the 
individual  evaluation  issues  and  criteria  already  developed  into 
overall  conclusions  on  operational  effectiveness  and  suitability 
for  employment  of  the  system.  Four  different  approaches  are 
presented.  Other  approaches  may  also  be  appropriate. 

3-62.  Primary  Measure  of  Effectiveness  (MOE) 

Some  systems  lend  themselves  to  use  of  a  primary  MOE  which  to  a 
large  extent  quantifies  operational  effectiveness  of  a  system 
which  is  sensitive  to  degradation  by  each  and  every  aspect  of 
effectiveness  and  suitability.  For  example,  suppose  that  a 
communication  system  to  provide  divisions  with  person-to-person 
communications  whether  stationary  or  on  the  move,  over  an  entire 
division  area,  with  interrupt  if  busy,  and  with  call-forwarding 
capabilities.  A  primary  MOE  can  be  defined  which  measures  number 
of  calls  successfully  completed  on  the  first  try  divided  by  the 
number  of  requirements  for  communication  whether  attempted  or 
not.  The  MOE  may  be  degraded  by  operator  error,  difficulty  of 
use,  RAM,  logistics  support,  MANPRINT,  or  system  performance 
under  a  wide  variety  of  conditions.  The  MOE  weighs  the  impact  of 
RAM,  logistics,  MANPRINT,  etc.  by  their  effect  on  the  system's 
ability  to  provide  division  communications. 


3-63.  Formal  decision  analysis 

When  systems  do  not  lend  themselves  to  use  of  a  primary  MOE,  and 
multiple  alternatives  exist  for  solving  an  operational  problem 
(competing  systems,  new  system  vs.  baseline)  and  have  several 
competing  attributes  (criteria  or  MOPs) ,  formal  decision  analysis 
techniques  provide  a  structured  way  to  weigh  relevant  aspects  and 
quantify  the  relative  worth  of  the  alternatives.  Multi-attribute 
decision  models  or  other  standard  operations  research  techniques 
can  be  effectively  used  to  rank  competing  alternatives  by  overall 
operational  effectiveness  and  suitability.  Examples  include 
"Dominance,"  "Maximin,"  "Maximax,"  Hierarchical  Additive 
Weighting,"  "Linear  Assignment,"  "ELECTRE,"  "TOPSIS,"  and 
"LINMAP . " 
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3-64.  Military  judgment 

The  least  formal  approach  is  the  use  of  military  judgment.  In 
this  approach  the  evaluator  uses  his  experience  to  determine, 
subjectively,  the  operational  effectiveness  and  suitability  of 
the  system.  This  approach  is  most  appropriate  on  systems  or 
concepts  that  have  singular  or  few  specific  capabilities  which 
have  clear  thresholds  of  acceptability.  The  more  complex  the 
system  and  the  more  capabilities  it  provides,  the  more  difficult 
it  is  to  weigh  good  and  bad  attributes  of  the  system  by  military 
judgment  alone  and  to  develop  overall  effectiveness  assessments. 

3-65.  Modeling  and  simulation 

Models  and  simulations  are  often  used  to  extend  OTE  results,  to 
explore  the  impact  of  deviations  from  requirements,  to  show  the 
expected  effectiveness  of  new  or  enhanced  capabilities,  or  to 
quantify  a  system's  operational  effectiveness  and  suitability. 
When  a  model  or  simulation  is  proposed  for  these  purposes,  it  is 
to  be  described  in  enough  detail  so  the  factors  played  and  the 
methodology  used  in  determining  the  outcome  of  the  model  can  be 
understood.  The  model  logic  should  support  inferences  about  the 
growth  of  soldier-weapon  interaction  over  the  life  cycle  of  the 
weapons  system.  Models  and  simulations  must  be  accredited  before 
use  in  OTE.  See  Part  One  for  use  of  modeling  and  simulation  in 
support  of  T&E  and  for  accreditation  procedures. 

3-66.  Development  of  the  system  evaluation  concept 
Detail  must  be  developed  on  proposed  analysis  methods  or  analytic 
considerations  that  are  planned  in  support  of  the  approaches  to 
system  evaluation.  The  evaluator  must  address  assumptions, 
limitations,  and  identified  risks  associated  with  the  analysis 
proposed.  OTE  primarily  intended  to  provide  early  operational 
experience  on  a  system,  to  verify  component  functionality,  or  to 
verify  corrections  to  identified  deficiencies  need  not  address 
all  aspects  of  system  operational  effectiveness  and  suitability. 
T&E  limitations  Bust  be  identified  and  considered  to  determine 
the  impact  of  t  se  limitations  on  the  evaluation.  Determine  why 
the  limitation  "ts  and  what  can  be  done  to  minimize  its 
impact . 


Section  IX 

Development  of  MOE  and  MOP 


3-67.  Measure  of  Effectiveness  (MOE) 

A  MOE  is  a  measure  that  expresses  the  extent  to  which  a  combat 
system  accomplishes  or  supports  a  military  mission. 

3-68.  Measure  of  Performance  (MOP) 
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A  MOP  is  a  parameter  that  expresses  the  extent  to  which  a  combat 
system  accomplishes  a  specific  performance  function.  In  general, 
higher  level  MOP  are  themselves  composed  of  either  lower  level 
MOP  or  data  requirements. 

3-69.  Data  requirement  (DR) 

A  DR  is  a  quantitative  or  qualitative  piece  of  information  that 
is  relevant  to  the  determination  or  categorization  of  one  or  more 
MOPs.  A  DR  can  consist  of  either  specific  test  measures  (e.g., 
start  time,  velocity,  position,  type  target)  or  arithmetically 
combined  measures  from  test  (e.g.,  elapsed  time,  calculated 
distance  between  points  a  and  b,  number  of  rounds  fired) .  A  data 
requirement  does  not  generally  involve  summary  statistics  (e.g., 
mean,  median,  percent) ,  which  are  usually  considered  lower  level 
MOPs. 

3-70.  Decomposition  of  operational  issues 

Evaluators  should  use  a  dendritic  process  for  developing  logic 
trees  and  work  breakdown  structures  for  decomposing  issues  into 
MOE,  MOE  into  MOP,  and  MOP  into  DR.  Factors  and  conditions  are 
integrated  and  necessary  event  dendritics  are  developed  to  both 
improve  and  structure  test  and  evaluation  planning. 

3-71.  Decomposition  of  criteria 

A  criterion  associated  with  an  issue  is  an  expression  of  the 
desired  level  of  accomplishment  of  the  system.  A  MOE  is  the 
quantification  of  the  extent  to  which  a  system  attains  the 
criterion.  The  MOE  (a  higher  level  measure  which  is  mission- 
oriented)  generally  encompasses  one  or  more  MOP.  MOE  may  be 
directly  measurable,  calculated  from  MOP,  or  based  on  military 
judgment. 

a.  For  example,  in  a  communications  network,  a. MOE  would  be 
the  degree  to  which  it  supports  division  command  and  control. 

The  MOP  might  be  completion  rate  or  availability  of  RF  links. 

b.  In  an  example  of  an  air  defense  system,  the  MOE  may  be  the 
degree  to  which  the  system  provides  protection  from  hostile  air 
attack.  The  MOP  might  be  the  ability  to  detect  or  engage. 

3-72.  Example  of  decomposition 

Operational  issues  define  the  relevant  questions  that  must  be 
answered  in  the  evaluation.  Answers  to  these  issues  come  from 
data  generated  from  many  sources,  the  most  important  of  which  is 
usually  the  operational  test.  As  a  vehicle  for  discussing  the 
development  of  MOPs  and  DRs,  an  example  issue,  associated  scope 
and  criteria  is  presented  in  figure  3—15.  The  example  issue, 
scope,  and  criteria  represent  a  typical  issue  and  criteria  and 
are  used  to  illustrate  the  process  used  to  develop  appropriate 
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MOPS  and  DRs.  The  criteria  presents  two  obvious  MOPs,  and  the 
scope  presents  considerations  relevant  to  factors  and  conditions 
that  need  to  be  addressed  when  answering  the  issue.  Close 
examination  of  the  issue  in  figure  3-15  shows  many  questions  not 
explicitly  stated,  which  need  to  be  answered  to  understand  the 
ability  of  the  system  to  locate  targets: 

(INSERT  FIGURE  3-15) 

a.  What  constitutes  a  target? 

b.  How  will  false  targets  be  handled? 

c.  What  constitutes  a  target  presentation? 

d.  What  constitutes  a  correct  detection? 

3-73.  Evaluation  planning  questions 

After  defining  terms,  questions  such  as  the  following,  become 
relevant  and  the  planning  methods  described  in  this  section  help 
identify  these  type  questions  and  lead  to  a  more  thorough  and 
well  structured  data  base  on  which  to  support  the  evaluation. 

a.  Are  any  of  the  functions  accomplished  by  the  system 
causing  deficiencies  in  the  time  or  accuracy  of  location? 

b.  Are  there  factors  or  conditions  which  lead  to  deficiencies 
in  time  or  accuracy  of  location? 

there  areas  where  training  or  man— machine  interface 
could  be  modified  to  improve  target  location? 

^^e  there  learning  or  other  trends  associated  with  target 
location  measures? 

3-74.  Evaluation  planning  objectives 

The  primary  planning  objective  is  to  understand  the  system 
response  to  the  environment  in  terms  relevant  to  the  issues  and 
to  support  quantification  of  how  well  the  matured  system  can 
function.  MOEs,  MOPs,  and  DRs  emerge  as  a  necessary  by-product 
of  this  planning.  Each  of  the  planning  methods  discussed  leads 
to  more  substantive  information  with  which  to  understand  the 
system  response.  Mutual  application  of  these  planning  methods 

j-gsult  in  a  more  credible  and  useful  evaluation  because  the 
evaluator  plans  not  only  for  the  estimation  of  system  capability: 
but  for  an  understanding  of  why  the  capability  is  as  it  is  and 
for 'estimating  how  that  capability  might  be  expected  to  change  as 
the  system  matures.  The  methods  suggested  also  help  in  the  early 
identification  of  required  instrumentation  and  data  organization. 
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3-75.  Functional  dendritics  ^ 

Evaluations  are  organized  by  the  identified  evaluation  issues  for 
the  system.  Criterion  statements  associated  with  the  issues 
typically  identify  the  primary  MOE.  The  evaluator  expands  and 
clarifies  the  primary  MOE  into  a  functional  dendritic  which 
covers  supporting  measures  of  performance  and  data  requirements 
appropriate  to  the  analysis  of  the  issue. 

a.  Development  of  this  dendritic  structure  begins  with  the 
evaluator  identifying  the  primary  functions  that  the  system 
performs  in  executing  its  mission.  These  primary  functions  ®re 
broken  out  into  secondary  (and  sometimes  tertiary)  functions  and 
into  MOP  which  quantify  the  performance  of  those  functions. 
Finally,  the  MOP  are  decomposed  into  the  set  of  data  requirements 
which  make  up  the  MOP. 

b.  For  the  example  issue  in  figure  3-15,  the  primary  mission 
of  providing  prioritized  target  information  is  quantified  in  the 
criterion  statement.  The  functions  that  support  successful 
execution  of  the  primary  mission  include  searching  the  target 
area,  detecting  targets  in  the  area  searched,  identifying  and 
classifying  as  red  or  blue  the  targets  detected,  prioritizing  the 
identified  targets,  locating  the  prioritized  targets,  and 
tracking  the  moving  targets  which  have  been  located. 

c.  To  search  a  target  area  effectively,  the  system  needs  to 
cover  the  search  area  and  do  it  efficiently.  How  does  one 
measure  coverage  and  efficiency?  How  do  inadequacies  in 
searching  the  target  area  effect  the  MOPs?  What  is  special  about 
the  system  which  is  relevant  to  searching  that  needs  to  be 
quantified?  What  makes  a  good  detection?  What  are  the 
capabilities  of  the  system  that  impact  or  aid  in  detecting?  How 
does  discrimination  between  true  and  false  targets  impact  the 
detection  of  true  targets?  How  does  the  success  of  the  search 
function  impact  the  detection  success?  How  is  classification 
success  determined?  How  is  it  impacted  by  the  validity  of  the 
target?  Is  efficiency  a  consideration?  What  is  correct 
prioritization?  How  is  it  measured?  How  do  undetected  targets 
effect  prioritization  success?  Dendritic  development  encourages 
this  type  of  questioning,  the  answers  to  which  strengthen  the 
evaluation  planning. 

d.  As  an  example,  the  target  location  issue  is  developed 
following  the  above  logic,  into  a  functional  dendritic  in  figure 
3-16.  The  dendritic  breaks  the  primary  mission,  providing 
prioritized  target  information,  into  lower  level  functions, 
supporting  measures  of  performance  and  finally  into  data 
requirements.  Factors  and  conditions  that  are  likely  to  impact 
the  measures  and  are  to  be  varied  in  test  or  analysis,  are  not 
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included  but  are  discussed  in  the  next  paragraph.  Each  end  point 
consists  of  measurable  data  which  is  traceable  to  the  issue 
through  the  dendritic.  This  approach  gives  a  reviewer  an 
organized  way  of  seeing  how  the  data  elements  were  derived,  and 
promotes  understanding  of  the  relationships  between  measures  and 
data  requirements. 


(INSERT  FIGURE  3-16) 

3-76.  Factors  and  conditions 

Factors  are  the  test  variables  identified  as  likely  to  affect 
test  event  outcome.  Conditions  are  the  discrete  values  that 
factors  assume  or  are  expected  to  assume.  For  example,  "range  at 
presentation"  is  often  regarded  as  a  factor  with  the  condition 
values  of  short  (less  than  1,500  meters),  medium  (1,500-2,500 
meters),  and  long  (greater  than  2,500  meters).  Likewise, 
"scenario"  is  often  regarded  as  a  factor  assuming  such  condition 
values  as  "offense,"  "defense,"  and  "tactical  road  march." 

Factors  represent  independent  variables  used  to  characterize  test 
events  and  are  used  to  categorize,  analyze,  and  evaluate  outcomes 
of  test  events. 

a.  Four  types  of  control  are  typically  applied  to  factors: 
held  constant,  systematically  varied,  tactically  varied,  or 
uncontrolled.  Personnel,  organization,  doctrine  and  tactics,  and 
logistical  support  are  normally  held  constant.  Factors 
controlled  by  the  tester  to  define  test  events  are  considered 
systematically  varied.  For  example,  because  they  are  controlled 
by  the  tester,  "offense"  and  "defense"  trials  are  treated  as 
systematically  varied  in  test  though  they  would  vary  tactically 
in  actual  combat.  When  not  directly  controlled  by  the  tester, 
engagement  conditions  (such  as  range,  firer  velocity,  target 
velocity,  target  aspect,  and  electro-optical  countermeasures)  are 
considered  to  be  tactically  varied.  Uncontrolled  factors 
generally  include  meteorological  conditions,  system  operational 
status,  and  player  attitudes. 

b.  The  set  of  factors  and  conditions  influen  ng  a  MOP  cannot 
be  practically  addressed  in  the  functional  dendritic  structure 
without  quickly  creating  an  unmanageable  number  of  branches.  The 
evaluator  must  create  and  use  a  logical  structure  to  control  the 
combinatoric  explosion  of  factors  and  conditions  that  threatens 
both  internal  and  external  validity.  Therefore,  the  evaluator 
identifies  the  factors  and  conditions  likely  to  have  meaningful 
impact  on  the  MOP  and  organizes  them  in  a  separate  chart  that 
serves  as  the  basis  for  identifying  situations  and  combinations 
of  situations  to  be  examined  in  studies,  models,  analyses  and 
tests;  that  provides  a  framework  within  which  the  evaluator  can 
identify  the  relative  frequency  of  occurrence  of  operationally 
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important  situations;  that  reflects  a  more  representative 
environment  by  ensuring  data  from  different  combinations  of 
situations  are  obtained  in  roughly  the  same  proportions  as 
expected  in  combat  or  are  analytically  weighted  to  give  summary 
measures  comparable  to  combat  expectations;  and  that  serves  to 
ensure  that  both  experimental  design  and  analysis  planning 
consider  influencing  variables  in  a  fitting  manner. 

c.  The  factors  and  conditions  chart  for  evaluation  planning 
is  developed  without  reference  to  whether  its  contents  are 
practical  to  test.  In  this  way  the  evaluator  is  most  likely  to 
consider  the  largest  set  of  relevant  influences  on  each 
particular  MOP.  Later,  in  planning  for  a  user  test,  a  more 
restricted  factors  and  conditions  chart  can  be  presented  address¬ 
ing  only  those  factors  and  conditions  to  be  varied  in  the  test. 

d.  Figure  3-17  builds  on  the  functional  dendritic  in  figure 
3-16  to  show  how  a  factors  and  conditions  chart  is  used  to 
identify  additional  data  requirements.  In  addition,  it  shows  the 
utility  of  the  chart  for  identifying  both  potential  sets  of  test 
circumstances  and  sets  of  circumstances  that  might  not  get 
exercised  in  test,  but  need  to  be  appropriately  compensated  for 
by  other  data  sources  or  addressed  through  analysis  and 
evaluation. 


(INSERT  FIGURE  3-17) 

e.  Based  in  part  on  the  analysis  concept,  the  evaluator 
determines  the  appropriate  factors  and  conditions,  together  with 
the  associated  degree  of  control,  and  presents  them  in  the  form 
of  a  tabular  list  (usually  grouped  by  type  of  control). 

Depending  on  the  extent  of  free  play  in  each  phase  of  the  planned 
test,  the  type  of  control  can  vary  by  phase  (e.g.,  systematic 
control  of  range  and  target  velocity  is  normally  required  in  a 
live-fire  phase  to  satisfy  range  constraints  but  not  during  dry 
fire  as  range  and  target  velocity  can  be  allowed  more  flexibility 
by  controlling  only  initial  trial  conditions  with  operations 
orders  and  initial  player  placement) . 

f.  The  tabular  list  typically  requires  footnotes  with 
accompanying  discussions  to  clarify  how  the  proposed  types  of 
control  will  ensure  that  appropriate  numbers  of  events  occur 
under  various  combinations  of  test  conditions.  Figure  3-17 
provides  a  typical  listing  of  factors,  types  of  control,  and 
conditions  for  a  typical  scenario. 

3-77.  Event  dendritics 

A  complementary  technique  useful  for  the  identification  and 
organization  of  required  data  is  the  use  of  event  dendritics. 
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Like  functional  dendritics,  event  dendritics  consist  of  a 
hierarchical  decomposition  of  system  functions  into  data  required 
for  analysis  and  evaluation.  However,  instead  of  decomposing 
these  functions  by  MOP  relevant  to  specific  issues  and  criteria, 
event  dendritics  decompose  these  functions  by  the  sequence  of 
"events”  performed  in  the  conduct  of  operational  missions. 

a.  An  "event"  is  an  opportunity  for  the  occurrence  of  an 
operationally  relevant  function  about  which  the  evaluator  or 
analyst  would  obtain  data.  Typical  events  might  be  designed  for 
target  detections,  firings,  target  passes,  movements,  handoffs, 
and  recoveries. 

b.  The  execution  of  functions  and  subfunctions  during  the 
conduct  of  an  operational  mission  generates  a  natural  hierarchy 
of  events  and  sub-events  (i.e.,  there  may  be  several  target 
passes  within  an  operational  mission,  several  detections  within  a 
target  pass  and  several  firings  for  each  detection) . 

c.  The  event  dendritic  starts  with  the  identification  of  the 
period  during  which  a  particular  series  of  events  might  occur. 

The  lower  levels  of  this  event  dendritic  will  consist  of  the 
lowest  level  actions  about  which  data  is  operationally  relevant. 
At  this  lowest  event  or  action  level,  the  evaluator  then 
identifies  the  set  of  data  relevant  to  the  conduct  of  the  action, 
which  might  influence  the  result,  which  is  necessary  to  give  a 
full  operational  description  of  the  event,  and  which  describes 
when,  where,  and  how  well  the  event  was  performed.  This  includes 
time  related  actions  associated  with  the  accomplishment  of  the 
event,  circumstances  around  which  the  event  occurred, 
environmental  conditions  that  could  influence  the  outcome, 
operator  actions,  responses  from  the  system,  and  system  status 
indicators,  any  of  which  might  correlate  to  the  system's 
performance  during  accomplishment  of  the  particular  event. 

d.  The  difference  bet  n  the  event  dendritic  and  the 
functional  dendritic  is  t  orientation;  measures  of  performance 
versus  events.  Functional  endritics  result  in  data  requirements 
that  support  the  development  of  identified  performance  measures. 
The  event  dendritic  results  in  data  requirements  which  describe 
the  conduct  of  operational  missions.  Both  are  required  for 
effective  evaluation  planning  and  will  benefit  later  test, 
analysis,  instrumentation  and  data  base  planning. 

e.  Figure  3-18  illustrates  an  event  dendritic  for  the  example 
issue.  It  is  predicated  on  the  existence  of  "engagement  periods" 
during  operating  the  system  which  are  recognizable  and  useful  for 
the  organization  of  data.  For  this  example,  an  engagement  period 
is  defined  to  begin  when  a  target  is  within  X  kilometers  and 
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intervisible  for  at  least  Y  seconds  and  to  end  when  either 
intervisibility  is  lost  for  z  seconds  or  the  target  passes  out  of 
range.  This  event  dendritic  complements  the  functional  dendritic 
and  the  associated  factors  and  conditions  chart.  The  event 
dendritic  results  in  the  identification  of  specific  engagement 
functions  that  encompass  the  subfunctions  relevant  to  detecting, 
acquiring,  and  firing  on  a  target  within  the  defined  start  and 
stop  times. 

(INSERT  FIGURE  3-18) 

3-78.  Data  requirements  planning 

The  end  product  of  the  functional  dendritic,  the  factors  and 
conditions  chart,  and  the  event  dendritic,  is  the 
needed  for  a  comprehensive  system  evaluation.  Each  of  the  three 
approaches  will  likely  need  expansion  based  on  the  results  of  the 
other  two.  Their  completion  is  an  iterative  process,  and  the 
products  produced  form  the  foundation  for  the  evaluation.  The 
perspectives  of  each  approach  differ  and,  as  the  examples  show, 
determine  a  complementary  but  different  set  of  data  requirements. 
Without  question,  these  examples  can  be  expanded  to  include  data 
requirements,  MOPs,  and  factors  not  shown.  The  examples  show, 
however,  the  thought  process  and  the  products  that  lead  to  a 
comprehensive  set  of  data  requirements  and  an  associated  data 
base  which  supports  an  effective  evaluation.  The  functional 
dendritic  and  the  factors  and  conditions  chart  contribute  to  the 
analysis  planning.  The  factors  and  conditions  chart  forms  the 
foundation  for  experimental  design  development,  and  the  event 
dendritic  forms  a  natural  organization  for  the  data. 

3-79.  MOP  dendritic  w  i. 

Figure  3-19  provides  a  generic  dendritic  for  MOP.  It  breaks 
mission  performance  into  six  major  functions  (target  location, 
mobility,  firepower,  C3,  computation,  and  productivity),  one  or 
more  of  which  may  be  appropriate  to  any  particular  system. 

Figure  3—19  suggests  only  one  of  the  many  possible  organizations 
for  performance  oriented  functions  and  is  presented  as  a  possible 
starting  point  for  mission  performance  planning.  Use  of  the 
functions  and  the  associated  MOPs  presented  in  the  example  will 
require  augmentation  and  adaptation  for  each  system.  For 
example,  a  sensor  system  might  have  as  its  primary  system  MOP, 
the  percentage  of  real  targets  accurately  located  in  a  30-minute 
scenario.  The  same  quantity  might  be  a  lower  level  MOP  for  a 
multi— function  combat  system  which  requires  not  only  target 
location,  but  weapon  system  function  and  quick  displacement  to  be 
effective. 


(INSERT  FIGURE  3-19) 
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Section  X 

Evaluation  and  Analys 


is  of  MOE  and  MOP 


A  MOP  measures  the  degree  a  system  accomplishes  ® 
function,  in  general,  a  MOP  may  be  composed  of  lower  level  MOPs 
or  data  requirements.  Each  MOP  can  be  categor^ed  as  an 
operational  Test  MOP  (OTMOP) ,  a  Modeling  MOP  ^  The 

Development  Test  MOP  (DTMOP)  to  describe  ‘^The* 

operational  tester  will  only  address  the  OTMOPs  .^5 

others  serve  to  describe  how  the  evaluator  will  fi 
from  sources  other  than  operational  test. 

The  evaluator  must  develop  the  logical  process  which 
to  use  to  resolve  the  issue.  He  decides  how 

identified  sources  will  be  integrated  and  will 

constraints  on  the  realism  or  the  _nalvses-^ 

be  treated.  He  develops  the  steps  used  to  interpret  analyse  , 

how  and  where  modeling,  simulation,  or  military  ^ 

used’  and  when  appropriate,  how  conclusions  on  individual 
c?uir!S^wIn  be'^intlgrated  to  resolve  the  issue  The  evaluator 
determines  the  comparisons  that  are 

that  will  be  made  and  ascertains  their  utility  to  the  evaluation. 

More’than^one^strategy^can  be  used  to  address  different  aspects 
o?  an  isSue?  and  occILonally  it  can  be  appropriate  to  use  more 
^ha^oie  strategy  to  address  the  same  aspect.  Discussion  of  each 
ascect  of  an  issue  is  to  include  factors,  conditions,  and 
oSratioLrscenarios  appropriate  to  the  evaluator's  Plan  to 
investigate  discrimination  between  the  systems,  organizations, 
methods  of  operation,  or  procedures.  Three  basic  comparative 
evalx  tion  strategies  are  typically  used. 

£  Comparison  of  new  or  competing  system  capability  to  the 
corr..  ponding  capability  in  the  system  being  replaced  (e.g., 
baseline) . 

b.  comparison  of  new  or  competing  system  to  a  predetermined 
standard. 

c.  Comparison  of  an  organization's  capability  with  and 
without  new  system. 

frameworK  within  which  data  for  all 
the  MOPS  will  be  analyzed.  The  evaluator  identifies  analytics 
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steps  planned  to  explore  and  understand  the  data,  integrate  data 
from  appropriate  sources,  summarize  or  re-express  the  data, 
estimate  parameters,  and  determine  trends  or  otherwise  explore 
the  data  in  a  manner  relevant  to  the  evaluation  of  the  data  set. 

3-84.  Analysis  concept 

The  analysis  concept  is  the  anticipated  framework  within  which 
data  for  the  issue  will  be  analyzed.  With  it,  the  evaluator 
identifies  the  analytical  steps  planned  to  explore  and  understand 
the  data,  integrates  data  from  appropriate  sources,  summarizes  or 
re-expresses  the  data,  estimates  parameters,  and  determines 
trends  or  otherwise  explores  the  data  in  a  manner  relevant  to  the 
evaluation  of  the  data  set  or  issue.  The  evaluator  identifies 
how  judgmental  criteria  and  weights  will  be  applied  in  the 
analysis  and  identifies  anticipated  graphical  or  arithmetical 
techniques  and  the  degree  to  which  the  analysis  will  be 
exploratory  (i.e.,  finding  out  what  the  data  are  trying  to  say) 
rather  than  confirmatory  (i.e.,  using  formal  statistical 
inference  to  answer  predetermined  questions) . 

a.  A  good  analysis  concept  serves  as  a  ”road  map”  for  the 
analyses  which  are  intended  to  identify  or  support  evaluative 
conclusions.  Like  any  road  map,  it  is  not  meant  to  be  rigidly 
followed  if  the  actual  data  or  other  circumstances  lead  to  a 
better  procedure.  The  use  of  decision  support  system  tools  is  an 
aid  in  developing  the  analysis  concept. 

b.  The  evaluator  identifies  the  specific  techniques 
appropriate  for  making  the  comparisons  or  estimates  called  for  in 
the  analysis  concept.  For  each  comparison  or  estimate,  the 
chosen  technique  must  be  planned  in  sufficient  detail  to 
establish  a  sound  analytic  treatment  for  the  operational  question 
being  asked.  Alternative  techniques  are  sometimes  appropriate, 
but  no  attempt  should  be  made  to  perform  each  and  every 
alternative  form  of  analysis. 

3-85.  Data  assumptions 

After  the  test,  actual  data  often  render  even  the  best  planned 
techniques  irrelevant  or  inappropriate.  The  evaluator  should 
identify  the  assumptions  associated  with  the  data,  the 
distributions,  and  the  use  of  proposed  analysis  techniques.  The 
extent  to  which  the  results  are  likely  to  be  sensitive  to 
deviations  from  the  assumptions,  especially  as  they  impact 
calculations  of  planned  confidence  intervals  and  significance 
statements  should  be  addressed  in  planning. 

3-86.  Data  independence 

The  independence  of  data  points  must  be  preserved.  The  many 
factors  which  typically  influence  the  utility  or  character  of  a 
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data  set  must  be  controlled.  The  evaluator  should  identify  known 
constraints  on  the  use  of  data  in  support  of  the  evaluation  and 
plan  to  handle  the  constraints  as  required.  Examples  of 
constraints  are:  data  from  a  model  which  does  not  play  realistic 
hostile  or  friendly  air  defense,  data  obtained  from  a  single 
environment,  data  from  immature  software,  logistics  data  limited 
to  realistic  maintenance  below  direct  support,  and  data  from 
crews  that  have  not  been  cross-trained.  The  evaluator  includes  a 
discussion  of  whether  the  constraints  will  be  handled 
judgmentally  or  with  formal  analysis  (specify  technique) ,  and 
clarifies  the  extent  to  which  the  impact  of  constraints  is  likely 
to  be  remedied. 

3-87.  Data  displays 

The  evaluator  proposes  the  data  displays,  tables,  figures,  or 
other  forms  of  presentation  appropriate  for  displaying  data, 
analyses,  or  results  of  analyses  in  reports. 


Section  XI 

Planning  for  Evaluation  Data 


3-88.  Data  sources  for  OTE 

While  the  OT  (EUT,  EUE,  LUT,  lOT,  or  FOT)  is  normally  the  primary 
source  of  data  for  the  operational  evaluation,  the  evaluator  roust 
consider  all  sources  of  data  available.  This  assu.  e.s  that 
(jata  are  considered  in  the  evaluation  and  prevents  duplication  of 
effort  in  the  scoping  of  the  OT.  If  the  user  test  duplicates 
other  efforts,  there  is  a  waste  of  resources.  The  evaluator  must 
look  at  all  sources  available  and  list  these  in  his  plan. 

a.  Possible  data  sources  other  than  the  user  test  include 

(1)  Other  user  test  being  conducted  (FDTE,  CEP,  CT) 

(2)  Development  testing  (DT) . 

(3)  Contractor  testing. 

(4)  Modeling  (to  include  the  COEA  modeling). 

(5)  Simulations. 

(6)  Market  surveys  and  investigations. 

(7)  Other  studies  and  analyses. 
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b.  All  sources  of  data  to  be  used  in  an  evaluation  or 
assessment  must  be  described  in  the  TEP. 

(1)  An  OT  listed  as  a  primary  or  secondary  data  source  for 
any  MOP  must  be  described  in  a  test  approach  paragraph.  The  OT 
to  be  planned  in  the  TEP  must  have  the  evaluator  test  approach 
and  concept  clearly  thought  out.  The  test  approach  varies  in 
scope  and  complexity  depending  on  the  size  and  sophistication  of 
the  anticipated  OT.  The  test  approach  provides  the  basis  for 
test  design  and  for  the  detailed  test  plan.  Derivation  of  a  test 
approach  is  discussed  in  detail  in  Section  XII,  below. 

(2)  Other  sources  of  data  are  investigated  by  the  evaluator 
to  determine  the  variety,  range,  and  quality  of  data  to  be 
anticipated.  The  evaluator  must  coordinate  with  the  personnel 
responsible  for  each  listed  data  source  in  the  planning  phase  to 
ensure  availability  of  required  data. 

(3)  User  tests  (other  than  those  OT  described  in  above), 

DT,  and  contractor  tests  will  be  described  in  sufficient  detail 
to  provide  understanding  of  the  test.  Models,  simulations, 
market  surveys  and  investigations,  studies,  and  analyses  will  be 
described  in  sufficient  detail  to  permit  both  understanding  of 
the  effort  and  the  value  and  context  of  the  data  derived  from  the 
source . 

3-89.  Development  of  the  data  source  matrix  (DSM) 

All  sources  of  data  to  be  used  in  an  evaluation  or  assessment 
must  be  described  in  the  TEP.  A  DSM  is  prepared. 

a.  Each  MOE  or  MOP  must  be  addressed  by  at  least  a  primary 
data  source.  A  primary  data  source  is  that  source  expected  to 
provide  all  the  essential  data  to  answer  the  measure.  Some 
measures  may  require  data  elements  from  more  than  one  source 
(i.e.,  OT  data  might  have  to  be  combined  with  live-fire  data  to 
compute  the  measure) .  In  these  cases,  both  data  sources  are 
considered  primary.  A  secondary  data  source  provides  data  to  be 
used  to  supplement  primary  data,  as  a  contingency  against  failure 
of  the  primary  source,  and  to  provide  additional  verification  of 
the  primary  data  source. 

b.  Any  test,  model,  simulation,  market  survey/ investigation, 
study,  or  analysis  listed  as  a  data  source  (primary  or  secondary) 
for  any  MOP  must  be  identified  by  the  evaluator  and  described. 

c.  The  evaluator  delineates  how  each  data  element  will  be 
utilized  in  the  evaluation.  The  evaluator  must  coordinate  with 
the  personnel  responsible  for  each  listed  data  source  in  the 
planning  phase  to  ensure  availability  of  required  data. 
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d.  The  DSM  is  the  matrix  of  the  sources  for  data  to  be  used 
in  the  evaluation  (for  example,  tests,  models,  analyses,  and 
studies)  matched  to  the  issues,  criteria,  and  MOE/MOP  which  each 
supports.  A  sample  DSM  is  shown  in  figure  3-20.  The  DSM  shows 
the  contribution  of  each  data  source  to  the  issue  and  to  each 
MOP. 


e.  The  matrix  cells  are  to  be  filled  with  "P"  or  "S"  to 
indicate  whether  the  data  source  will  provide  a  "primary”  or 
"secondary"  contribution  to  the  evaluation.  Matrix  cells  are 
left  blank  for  data  sources  which  are  neither  primary  or 
secondary.  Each  MOP  must  have  at  least  a  primary  data  source. 

f.  Based  on  the  requirements  of  the  DSM  for  data,  the 
evaluator  can  scope  the  size  of  the  test  and  the  extent  of 
evaluator  initiated  modelling  and  simulation.  The  evaluator  also 
can  assure  himself  that  every  MOE/MOP  will  be  answered. 

(INSERT  FIGURE  3-20) 


Section  XII 

Derivation  of  the  User  Test  Approach 


3-90.  The  operational  test  design  concept  (OTDC) 

The  OTDC  varies  in  scope  and  complexity  depending  on  the  level  of 
sophistication  of  the  anticipated  test.  It  provides  a  basis  for 
the  test  design  in  the  TEP  (where  details  of  the  test  design  are 
documented)  and  for  the  Detailed  Test  Plan  (where  the  mechanisms 
for  carrying  out  the  TEP  are  specified) .  The  OTDC  requires 
enough  detail  to  ensure  that  a  test  designed  lAW  the  concept  will 
provide  the  data  necessary  to  answer  the  evaluation  issues.  In 
addition,  it  requires  enough  flexibility  to  permit  evolution  as 
test  planning  progresses  and  graceful  degradation  in  the  presence 
of  unplanned  test  delays  or  shortfa}  The  OTDC  identifies  the 

following: 

a.  The  types  of  tactical  scenarios  to  be  conducted,  the 
degree  of  operational  realism  (to  include  threat  portrayal) 
required  to  answer  the  evaluation  issues,  and  the  types  of  test 
events  about  which  data  are  required. 

b.  A  list  of  the  factors  and  conditions  likely  to  effect 
outcomes  of  test  events,  and  an  approach  for  controlling  test 
factors  to  ensure  that  various  types  of  events  occur  under 
appropriate  combinations  of  test  conditions . 
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c.  A  framework  for  grouping  combinations  of  test  conditions 
into  trials,  vignettes,  missions,  and  phases  for  test  conduct  and 
an  assessment  of  the  sample  sizes  required  to  control  the  risks 
associated  with  anticipated  analyses. 

3-91.  OTMOP 

OTMOP  form  the  basis  for  the  OT  and  the  test  design  in  the  TEP. 
The  OT  objective  is  to  answer  the  OTMOP  derived  from  OIG.  Based 
on  the  OIC  in  paragraph  2.2  of  the  TEP  and  the  DSM,  the  evaluator 
identifies  OTMOP.  OTMOP  deal  with  questions  best  answered  in  OT 
and  which  cannot  be  adequately  answered  by  DT,  contractor  test, 
modeling,  simulations,  studies  or  other  data  sources. 

3-92.  The  scope  of  OT 

The  scope  identifies  the  types  of  test  scenarios  and  test  events 
that  must  answer  the  test  issues.  The  level  of  detail  includes 
the  approximate  length  of  various  test  scenarios;  the  number  of 
test  items  to  be  exercised  in  each;  the  type  and  size  of  player, 
support,  and  threat  units;  and  the  test  environment  for  maneuver 
area,  terrain,  vegetation,  and  security  required.  Additional 
details  to  be  specified  in  this  paragraph  concern  the  degree  of 
operational  realism  required. 

3-93.  OT  simulation  of  actual  combat 

The  degree  of  operational  realism  required  depends  on  the  types 
of  scenarios,  types  of  test  events,  and  combinations  of  test 
conditions  required  to  answer  test  issues. 

a.  The  most  realistic  simulations  occur  in  two-sided  battles 
supported  by  Real  Time  Casualty  Assessment  (RTCA) .  In  such 
simulations,  the  events  of  primary  interest  are  engagements 
between  pairs  of  players.  Initial  battle  conditions  are 
determined  before  the  start  of  each  test  trial.  RTCA  is  used  to 
shape  the  test  battle  by  encouraging  sequences  of  individual 
engagement  conditions  representative  of  combat  under  the 
specified  initial  conditions.  Thus,  initial  trial  conditions  are 
systematically  varied  but  individual  engagement  conditions 
develop  tactically  and  cannot  be  guaranteed.  Such  testing  is 
expensive  and  time  consuming  due  to  the  elaborate  instrumentation 
and  instrumentation  checkout  requirements,  and  the  necessity  for 
ensuring  player  uncertainty  on  both  sides  of  the  simulated 
battle. 


b.  Important  cost  savings  and  increases  in  test  efficiency 
are  possible  if  less  realism  is  acceptable.  Often  field  training 
or  command  post  exercises  are  appropriate  for  testing  a  system  or 
concept.  In  these  cases,  the  duration  of  the  exercise  should  be 
adequate  to  address  issues  such  as  continuity  of  operations  or 
soldier  stress  and  fatigue,  when  appropriate  to  the  evaluation. 
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c.  In  some  cases,  realism  is  required  for  only  one  side  of 
the  engagements.  The  other  side  can  be  played  according  to  a 
pre-planned  script.  For  example,  if  the  primary  test  issue 
involves  platoon  engagement  capability  against  certain  threat 
presentations,  carefully  scripted  threat  presentations  to  a 
tactically  deployed  platoon  produce  a  more  precise  answer  than  a 
series  of  two-sided  RTCA  engagements.  In  this  case,  only  threat 
presentations  to  the  platoon  need  be  realistic.  Under  some  such 
circumstances ,* combinations  of  analytic  modeling  with  one  sided 
operational  testing  can  produce  more  extensive  and  believable 
results  with  less  expense  than  by  testing  alone. 

d.  In  other  cases,  realism  about  simulation  of  combat  itself 
is  of  lesser  importance.  Testing  of  centralized  systems  involved 
with  intelligence  gathering,  communications,  or  disruption  of 
communications  generally  requires  realism  primarily  in  the 
electronic  environment.  Although  emplacements,  displacements, 
and  simulated  field  conditions  are  required,  realistic 
operational  stress  on  both  man  and  machine  is  due  primarily  to 
types,  numbers,  and  timing  of  messages  or  digital  traffic. 

Factors  and  conditions  to  be  manipulated  primarily  involve 
message  content,  timing,  link  lengths  or  system  configuration 
(number  of  nodes,  nets,  and  paths).  Although  many  personnel 
(possibly  augmented  by  computer  driven  simulators)  may  be 
required  to  generate  a  realistic  electronic  environment 
representative  of  combat,  it  will  seldom  be  necessary  to  have 
large  numbers  of  real  people  and  equipment  involved  in  simulated 
combat . 

3-94.  Test  factors  and  conditions 

Factors  are  the  test  variables  identified  as  likely  to  effect 
test  event  outcome.  Conditions  are  the  discrete  values  that 
factors  assume  or  are  expected  to  assume.  For  example,  "range  at 
presentation"  is  often  regarded  as  a  factor  with  the  condition 
values  of  shor'  <1500  m) ,  medium  (1500-2500  m) ,  and  long  (>2500 
m) .  Likewise  scenario"  is  often  regarded  as  a  factor  assuming 
the  condition  ues  "offense,"  "defense,"  and  "tactical  road 
march . " 


a.  Factors  represent  variables  used  to  characterize  test 
events  and  are  used  to  categorize,  analyze,  and  evaluate  outcomes 
of  test  events.  Four  types  of  control  are  typically  applied  to 
factors:  fixed,  systematically  varied,  tactically  varied,  or 

uncontrolled.  Personnel,  organization,  doctrine  and  tactics,  and 
logistics  support  are  normally  "fixed"  (or  "held  constant"). 
Factors  controlled  by  the  tester  to  define  test  events  are 
considered  to  be  systematically  varied.  For  example,  because 
they  are  controlled  by  the  tester,  "offense"  and  "defense"  trials 
are  treated  as  systematically  varied  in  test  though  they  would 
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vary  tactically  in  actual  combat.  When  not  directly  controlled 
by  the  tester,  engagement  conditions  (e.g.,  range,  firer 
velocity,  target  velocity,  target  aspect,  electro-optical 
countermeasures)  are  considered  to  be  tactically  varied. 
Uncontrolled  factors  generally  include  meteorological  conditions, 
system  operational  status  and  player  attitude. 

b.  Based  in  part  on  the  analysis  concept,  the  evaluator 
determines  the  appropriate  factors  and  conditions,  together  with 
the  associated  degree  of  control,  and  presents  them  in  the  form 
of  a  tabular  list  (usually  grouped  by  type  of  control). 

Depending  on  the  extent  of  free  play  in  each  phase  of  the  planned 
test,  the  type  of  control  can  vary  by  phase.  For  example, 
systematic  control  of  range  and  target  velocity  is  normally 
reguired  in  a  live-fire  phase  to  satisfy  range  constraints,  but 
not  in  a  dry  fire  field  exercise.  During  dry  fire,  range  and 
target  velocity  can  be  allowed  more  flexibility  by  controlling 
only  initial  trial  conditions  with  operations  orders  and  initial 
player  placement.  The  tabular  list  typically  requires  footnotes 
with  accompanying  discussions  to  clarify  how  the  proposed  types 
of  control  will  ensure  that  appropriate  numbers  of  events  occur 
under  various  combinations  of  test  conditions. 

3-95.  Test  design  matrix (es) 

The  evaluator  controls  the  risk  associated  with  anticipated 
analyses  by  identifying  the  number  of  valid  events  to  be 
conducted  under  specific  combinations  of  test  conditions  and  by 
identifying  the  considerations  (e.g.,  order,  pairing)  about  the 
conduct  of  those  events  to  support  anticipated  analyses. 

a.  The  development  of  sound  interval  estimation  techniques  is 
critical.  The  decision  the  evaluator  will  make  is  a  decision 
about  the  truth  or  falsity  of  a  pre-designed  statistical 
hypotheses.  Lack  of  detailed  planning  in  this  part  of  the  test 
concept  may  lead  to  inferring  no  statistically  significant 
difference  was  demonstrated  by  the  test  when  a  difference 
actually  exists. 

b.  Once  the  required  types  of  test  events  and  scenarios  have 
been  identified  in  the  scope  and  appropriate  factors  and 
conditions  have  been  formulated,  the  evaluator  develops  a 
framework  for  grouping  combinations  of  test  conditions  into 
"trials”  and/or  "missions"  and/or  "phases."  He  determines  how 
many  test  events  of  each  type  are  likely  to  occur  in  each  type 
trial  or  mission  and  how  many  trials  or  missions  are  possible 
during  the  available  test  days  or  weeks. 

c.  On  this  basis,  he  estimates  for  tests  of  various  lengths 
and  for  alternative  formulations  of  test  elements,  the  aggregate 
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sample  sizes  likely  to  occur  under  each  combination  of  factors 
and  conditions.  Often  the  aggregate  sample  size  is  determined  by 
the  time  period  and  number  of  systems  available  for  test.  In 
this  case,  statistical  procedures  can  only  be  used  to  estimate 
the  risks  involved  with  alternative  allocations  of  resources. 

d.  When  there  is  more  flexibility  in  scenario  duration, 
number  of  trial  repetitions,  number  of  systems,  and  other 
resources  available  for  test,  more  rigorous  statistical 
procedures  can  be  used  to  determine  sample  sizes  that  will  reduce 
the  risks  to  specified  levels.  Once  sample  sizes  are  determined, 
they  are  tabulated  as  matrixes  showing  numbers  of  valid  test 
events  to  be  conducted  under  various  combinations  of  test 
conditions. 

3-96.  Sample  sizes  in  OT 

The  analyst  must  balance  the  requirements  for  statistical 
significance  in  each  of  the  cells  of  his  matrixes  with  the 
requirements  that  the  system  follows  the  OMS/MP  in  test. 

a.  Rigorous  statistical  sample  sizing  can  only  be  done 
precisely  when  two  conditions  are  satisfied.  First,  only  a  few 
combinations  of  test  conditions  can  be  involved.  Second,  a  good 
estimate  of  test  variability  must  be  available  for  the  MOP  under 
consideration.  When  these  considerations  are  met,  simple  models 
based  on  binomial,  normal  or  other  common  distributions  are  used 
to  determine  the  minimum  sample  size  requirements.  NOTE:  The 
text  Experimental  Statistics  by  M.  G.  Natrella  (originally 
printed  as  a  series  of  pamphlets  by  AMC)  is  a  good  reference  for 
such  sample  sizing  techniques. 

b.  Before  performing  statistical  sample  sizing,  answers  to 
the  following  questions  are  required: 

(1)  Is  the  observed  value  for  the  OTMOP  to  be  compared  to  a 
prescribed  value  or  to  another  observed  value?  Determine  whether 
the  comparison  value  is  assumed  to  be  measured  without  error  or 
is  a  value  from  observations  of  a  baseline  or  alternative  which 
is  itself  subject  to  measurement  error  because  comparisons  to 
observed  values  require  larger  sample  sizes. 

(2)  Is  the  comparison  one-  or  two-sided?  Examples  of 
one-sided  comparison  outcomes  are:  ’'significantly  larger  than 
prescribed  value"  versus  "not  significantly  larger  than 
prescribed  value,"  or  "significantly  less  than  the  baseline" 
versus  "not  significantly  less  than  the  baseline."  Examples  of 
two-sided  comparison  outcomes  are:  "significantly  larger  than 
prescribed  value"  versus  "significantly  smaller  than  prescribed 
value"  versus  "not  significantly  different  from  the  prescribed 
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value,”  or  ”signif icantly  less  than  the  baseline”  versus 
••significantly  more  than  the  baseline”  versus  ”not  significantly 
different  from  the  baseline.”  Two-sided  comparisons  require 
larger  sample  sizes. 

(3)  How  large  must  the  difference  between  the  tested  item 
and  the  prescribed  or  observed  value  be  before  detecting  that  the 
difference  is  significant?  Detecting  smaller  differences 
requires  larger  sample  sizes. 

(4)  What  risks  are  tolerable?  Two  risk  levels  require 
specification.  One  risk  level  corresponds  to  a  failure  to  detect 
a  difference  wht'!  an  important  difference  really  exists.  The 
other  risk  level  ::orresponds  to  declaring  that  a  difference 
exists  when  it  does  not.  Smaller  levels  of  acceptable  risk 
require  larger  sample  sizes. 

c.  OT  rarely  satisfies  the  two  conditions  necessary  for 
precise  sample  sizing  because  operational  testing  typically 
involves  many  combinations  of  test  conditions  and  estimates  of 
OTMOP  variability  during  test  are  vague  at  best. 

d.  Data  to  compute  several  OTMOPs  with  different  variability 
are  obtained  simultaneously.  Acceptable  or  desired  performance 
requirements  often  vary  as  test  conditions  vary ,  but  allowable 
limits  to  this  variation  are  seldom  specified.  The  magnitude  of 
differences  which  are  important  to  detect  is  seldom  specified 
precisely.  Nevertheless,  statistical  sample  sizing  is  an 
essential  part  of  operational  test  planning.  Estimates  from 
sample  size  calculations  (even  when  underlying  statistical 
assumptions  are  violated)  can  provide  rough  assessments  of 
differences  likely  to  be  detected  in  operational  test  at  the 
associated  risk  levels. 

e.  When  OT  requirements  are  complicated,  application  of  more 
sophisticated  statistical  considerations  can  clarify  impacts  of 
trade-offs  between  various  test  design  alternatives  in  sample 
size  and  risk  levels.  These  considerations  deal  with  statistical 
design  and  analysis  of  experiments,  the  branch  of  statistics 
which  studies  techniques  for  extracting  the  maximum  amount  of 
information  from  an  ”experiment. •• 

f.  Since  most  research  in  the  area  of  sample  sizes  has  dealt 
with  ”experimentation,”  which  is  much  more  structured  than 
operational  testing,  caution  is  required  when  applying  notions 
from  experimental  design  to  the  design  of  operational  tests. 
However,  some  application  of  experimental  design  is  useful  for 
efficient  operational  test  planning.  The  use  of  quasi- 
experimental  design  and  analysis  for  field  settings  may  be 
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considered  useful  by  the  evaluator.  The  use  of  non-parametric 
statistics  may  be  appropriate  for  some  operational  test  samples. 

g.  Simple  experimental  design  notions  come  into  play  because 
alternatives  in  test  element  formulation  and  test  length  cannot 
be  compared  solely  on  the  basis  of  sample  size.  Variability  in 
OTMOPs  to  be  used  in  performing  comparisons  is  also  of  crucial 
importance.  For  a  given  sample  size,  experimental  design 
techniques  can  reduce  variability  in  the  comparisons  of  greatest 
interest,  by  increasing  similarity  of  test  conditions  (e.g.,  by 
using  paired  comparisons  or  blocking) ,  or  by  carefully 
controlling  differences  between  test  conditions  (e.g.,  by  using 
Latin  Squares  or  other  fractional  factorial  designs) . 

h.  In  addition  to  addressing  variability,  notions  from 
experimental  design  are  necessary  to  address  possible  bias  and 
confounding,  as  well  as  feasibility  of  obtaining  specified  sample 
sizes.  Potential  bias  or  confounding  occurs  when  possibly 
influential  factors  (either  controlled  factors  or  uncontrolled 
nuisance  factors)  are  not  evenly  applied.  Experimental  design 
techniques  can  minimize  or  adjust  for  such  confounding  and 
establish  what  combinations  of  sample  sizes  are  mathematically 
possible  using  a  specified  design  framework. 

i.  Because  of  the  frequency  with  which  instrumentation  or 
operational  circumstances  cause  loss  of  data,  caution  is^ 
recommended  in  the  use  of  exotic  experimental  designs  which  rely 
heavily  on  balance  or  exact  numbers  of  valid  samples. 

j .  The  experimental  design  techniques  that  have  proven  to  be 
most  successful  involve  designing  in  small  ”blocks”  so  that  test 
events  to  be  compared  occur  close  together  in  time  or  space.  The 
simplest  such  designs  are  paired  comparisons  involving 
"side-by-side”  tests  of  a  candidate  and  a  baseline.  More 
elaborate  designs  involve  testing  several  candidates  under 
somewhat  different  conditions  at  nearly  the  same  “ime  or  nearly 
the  same  place  and  rotating  candidates  and  cone  ns  through 
appropriate  locations  and  combinations  of  condi  iS  over  a 
period  of  days  or  weeks. 

k.  Such  techniques  can  be  used  to  build  up  a  test  design  on  a 
day-by-day  or  week-by-week  basis  so  added  cells  complement 
existing  cells  rather  than  attempting  to  replicate  them.  Each 
group  of  added  cells  not  only  examines  additional  combinations  of 
conditions  but  reduces  overall  potential  for  confounding. 

Designs  constructed  in  this  incremental  manner  not  only  fit 
naturally  with  realities  of  test  conduct  but  also  permit 
systematic  assessment  of  alternative  allocations  of  test  time. 
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Section  XIII 
Evaluation  Data  Base 


3-97.  Known  data  support  requirements 

The  evaluator  needs  to  identify  known  instrumentation  and  data 
processing  requirements  that  are  not  immediately  obtainable 
off-the-shelf.  The  focus  here  is  to  list  any  specialized 
instrumentation  or  instrumentation  interfaces,  simulators, 
computer  hardware  or  software  that  the  evaluation  planning 
process  has  identified  as  likely  to  be  necessary  for  capturing 
and  analyzing  required  data.  The  evaluator  identifies  whether 
there  is  a  requirement  for  a  Data  Authentication  Group  (DAG) . 
Additional  information  on  the  DAG  is  provided  in  chapter  5. 

3-98.  Requirements  for  the  evaluation  data  base 
The  evaluator  must  develop  and  document  a  test  data  base  file 
structure  that  will  enable  him  to  analyze  test  data  in  a  thorough 
and  timely  manner,  and  to  integrate  it  with  data  from  other 
sources.  Known  data  elements  need  to  be  organized  according  to 
this  file  structure  and  definitions  of  data  elements  need  to  be 
provided.  The  evaluator  data  base  may  contain  up  to  level  seven 
data.  The  test  data  base  will  usually  contain  level  three  data 
and  will  not  contain  data  above  level  4.  See  figure  3-21  for 
definitions  and  examples  of  levels  of  data. 

(INSERT  FIGURE  3-21) 


a.  Evaluator  specification  of  data  base  structure  and  content 
must  ensure  that  data  base  descriptions  of  test  events  are 
sufficient  to  permit  flexible,  thorough,  and  timely  analyses  even 
where  events  occur  which  may  be  unanticipated.  Organizing  test 
data  elements  according  to  what  is  being  described  (e.g.,  test 
events,  sub-event,  test  items,  weather  events)  usually  provides 
the  most  effective  file  structure  for  flexible  and  timely 
analysis.  The  event  dendritic  approach  discussed  above  will 
result  in  such  a  data  structure.  Figure  3-22  provides  a  typical 
data  base  structure.  The  level  of  detail  in  the  figure  is 
suitable  for  a  TEP.  With  appropriate  arrows  added,  the  diagram 
in  figure  3-22  describes  relationships  between  files. 

(INSERT  FIGURE  3-22) 


b.  The  TEP  description  of  the  test  data  base  must  include 
identification  of  the  required  files  to  include  specification  of 
the  files  required,  and  labelling  each  file  with  a  short 
description  of  the  type  of  data  to  be  stored  in  that  file,  and  an 
indication  for  each  file  whether  it  is  to  be  automated. 
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c.  The  TEP  roust  provide  a  description  of  relationships 
between  required  files  using  hierarchical  or  network  diagrams  and 
identification  of  key  elements  (e.g.,  trial  number,  item 
identification,  date,  time,  crew  number)  that  are  common  to 
several  files  that  define  the  relationships  between  files.  NOTE; 
It  is  usually  important  to  relate  HANPRINT  data  to  appropriate 
effectiveness  and  suitability  data. 

d.  The  TEP  must  define  data  elements  in  each  file  providing  a 
list  of  known  data  elements  defined  to  preclude  any  ambiguity  or 
misunderstanding  on  the  part  of  the  tester. 

e.  Figure  3-23  provides  an  example  of  a  data  base  record 
structure.  The  level  of  detail  is  suitable  for  a  file 
description  in  an  TEP. 


(INSERT  FIGURE  3-23) 

f.  The  data  base  structure  to  be  provided  later  in  chapter  3 
of  the  TEP  expands  that  in  chapter  2  by  adding  files  or  data 
elements  necessary  for  test  control  or  data  quality  control,  or 
by  refining  existing  files  and  data  requirements  considering 
evolving  test  planning. 


Section  XIV 

Documentation  of  the  TEP  Evaluation  Plan  (TEP  Chapter  2) 


3-99.  Preparation  of  chapter  2  of  the  TEP 

After  completion  of  the  above  steps,  the  evaluator  documents  his 
evaluation  planning  in  chapter  2  of  the  TEP.  Chapter  2  is  the 
Evaluation  Plan.  Chapter  2  is  prepared  by  either  the  evaluator 
or  the  tester  depending  on  the  type  of  test  or  evaluation.  See 
figure  3-3  for  allocation  of  responsibilities  for  the  preparation 
of  this  chapter.  If  the  apter  is  prepared  by  the  evaluator,  it 
is  called  the  Evaluation  .n.  If  the  chapter  is  prepared  by  the 
tester,  it  is  called  the  .  alysis  Plan.  Paragraph  side  headings 
will  use  the  evaluation  terminology  if  prepared  by  the  evaluator 
and  analysis  terminology  if  prepared  by  the  tester.  When  the 
tester  prepares  an  analysis  plan  in  this  chapter,  the  effort  is 
simplified  and  abbreviated. 

a.  In  this  chapter  of  the  plan,  the  evaluator  (or  tester) 
presents  the  methodology  by  which  results  of  testing  and  other 
data  sources  will  address  the  individual  QIC  in  chapter  1  of  the 
TEP  and  be  consolidated  into  overall  conclusions  or  predictions 
on  the  operational  effectiveness  and  suitability  for  employment 
of  the  system  being  evaluated.  Details  are  to  be  given  on 
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b.  Aggregation  of  operational  suitability  shows  the 
methodology  by  which  the  answers  to  the  issue  questions  will  be 
further  analyzed  and  consolidated  to  permit  conclusions  or 
predictions/assessments  to  be  drawn  on  the  overall  operational 
suitability  of  the  system. 

3-101.  Operational  issues 

This  paragraph  lists  in  turn  each  of  the  operational  issues  and 
associated  criteria  stated  in  chapter  1  of  the  TEP.  Each 
subparagraph  will  contain  an  issue  statement  taken  in  turn  from 
the  issue  statements  in  chapter  1.  The  issues  are  stated  in  the 
form  of  a  question  for  which  a  response  will  make  a  significant 
contribution  to  the  decision  making  process.  Scope  and  rationale 
need  not  be  restated  here  as  they  are  already  provided  in  chapter 
1  and  restatement  would  be  unnecessary  duplication. 

a.  Operational  effectiveness  issues  deal  with  the  overall 
degree  of  mission  accomplishment  when  the  system  is  used  by 
representative  personnel  in  the  environment  planned  or  expected 
for  its  operational  employment  considering  organization, 
doctrine,  tactics,  survivability,  vulnerability,  and  threat 
(including  countermeasures,  nuclear,  and  chemical  or  biological 
threats) .  The  "system”  includes  the  hardware  (to  include  GFE) , 
software,  personnel,  and  operating  and  support  procedures. 

b.  Operational  suitability  issues  address  the  degree  to  which 
a  system  can  be  satisfactorily  placed  in  field  use  with 
consideration  given  to  RAM,  compatibility  and  interoperability, 
transportability,  wartime  usage  rates,  safety  and  human  factors, 
manpower  requirements,  logistics  supportability,  documentation, 
and  training  requirements.  Important  considerations  include,  for 
example,  the  understanding  of  what  adjustments  must  be  made  in 
logistical  support  to  service  the  new  system  without  disrupting 
service  given  to  existing  systems;  whether  existing  soldier 
skills,  characteristics,  IQ,  etc.  are  adequate  to  successfully 
employ  and  maintain  the  new  system;  or  whether  new  skills  nd 
talents  are  needed. 

3-102.  Operational  criteria 

Each  issue  subparagraph  is  further  divided  into  sub-subparagraphs 
for  each  criterion  statement  taken  in  turn  from  the  criteria 
statements  for  the  issues  in  chapter  1.  Criteria  which  were  a 
part  of  the  original  COIC  set  will  always  have  both  a  measure 
(parameter)  and  a  threshold  (value) .  Criteria  developed  by  the 
evaluator  may  have  only  a  measure.  In  those  cases  where  the 
evaluator  criterion  lacks  a  threshold  (value) ,  the  criterion  will 
be  labeled  "investigative"  and  only  the  measure  (parameter) 
specified. 
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3-103.  MOE/MOP 

sub-sub- 

Each  criterion  must  hav^  derived  to  measure  a  criterion 

prepares  chap2e?  ^SeJe  te^Je^ 

may  find  it  necessary  to  use^both  Soi^^  be  only  mop.  Evaluators 
must  define  the  individual  MOE/MOP  A*2fto**°^*  Paragraph 

to  which  a  combat  system  accomolifihFbe^**®^  expresses  the  extent 
the  DSM,  each  MOP  is  cateaorT^oi  ®  ®  specific  function,  m 
(^OP) ,  or  a  DTMOP)  to  describe\ho  0™°^^  a  modeling  MOP 
will  only  address  the  OTMOP^in  Iht  ^®®ter 

serve  to  describe  how  the  evain-J?  The  others 

sources  other  than  operational  test  5^^^  data  voids  from 

section  XVI  detail  derivatiSS^of  ®!ftions  ix  and  X  and 
MOE  or  MOP  will  be  described  as  folliwsf  treatment  of  each 

pro«ss™l  2^"iSd\o®milsure  «'"  l°9ical 

how  the  data  from  tie  ilenUfted  describes 

thP  constraints  on  the  reaUsm^or  ?h  ^”^®9rated  and 

the  data  will  be  treated.  He  described  completeness  of 

used  to  interpret  analyses-  W  Sk  will  be 

or  military  judgment  will  be^sed^^nd^^wh®®"^®^^"^'  simulation, 
conclusions  will  be  inteoratoH  appropriate,  how 

analyst  specifies  tt"M^pa^UoJs  ori:;****®  ^he 

basis  Of  the  analysis  and  tJe  es?l»a^.f  the 

describes  the  utility  of  th#»  will  be  made  and 

analysis.  Three  basic  comSirlS  estimates  to  the 

comparison  of  the  new  or  COTpetiSa  sJst^2^®®  ere  typically  used 
corresponding  capability  in  the  the 

comparison  of  the  new  o^  competinJ  svst^^?®  replaced  (baseline) , 
Standard,  or  comparison  of  a^mili?a>-^^^®®  ®  Predetermined 

with  and  without  the  new  system^  capability 

gained  by  looking  at  the  information  is 

different  conditions  (e.gf  dav  ve  «?^k5?®  "®'I'  system  under 
analytic  design  for  the  measure  is  *  *  .a  Discussion  of  the 

analyst°s^plan\o^iJvestigate°d”*^^°® • °P^^®t2^to^the 
organizations,  methods  of%erfMSnrorpwced«e2®"  systems, 

fram;worri?Jh?n'’iMlh“he  dna“w?!!'=£^‘’“:?  anticipated 

serves  as  a  -road  map"  for  the  a^alv.^  ’’*'1®  section 

Identify  or  support  evaluLive  “niluIionS^''^'”'®  intended  to 
^nparametric  techniques  should  be  sMcifiad  *^®tametric  and 
be  discussed,  and  statistical  assumptions  should 

specified.  Group  the  mIe  and  MS“S”iK"an!?; 
each  comparison  or  estimate,  discuss  choSer?echi?qSirif  ® 
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sufficient  detail  to  establish  that  there  is  a  sound  analytic 
treatment  for  the  measure.  Limited  discussion  of  alternative 
techniques  is  sometimes  appropriate,  but  no  attempt  is  made  to 
provide  an  exhaustive  list  or  comprehensive  discussion  of 
alternatives.  The  actual  data  often  render  even  the  best  planned 
techniques  irrelevant  or  inappropriate.  The  analyst: 

(1)  Identifies  the  analytical  steps  planned  to  explore  and 
understand  the  data,  integrate  data  from  appropriate  sources, 
summarize  or  re-express  the  data,  estimate  parameters,  and 
determine  trends  or  otherwise  explore  the  data  in  a  manner 
relevant  to  the  evaluation  of  the  data  set. 

(2)  Identifies  how  judgmental  criteria  and  weights  will  be 
applied  in  the  analysis. 

(3)  Identifies  anticipated  graphical  or  arithmetical 
techniques  and  the  degree  to  which  the  analysis  will  be 
exploratory  (finding  out  what  the  data  are  trying  to  say)  rather 
than  confirmatory  (using  formal  statistical  inference  to  answer 
predetermined  questions) . 

(4)  Identifies  the  specific  techniques  appropriate  for 
making  the  comparisons  or  estimates  identified  in  the  evaluation 
concept . 

(5)  Identifies  the  assumptions  associated  with  the  data, 
the  distributions,  and  the  use  of  proposed  analysis  techniques. 

(6)  Discusses  the  extent  to  which  the  results  are  likely  to 
be  sensitive  to  deviations  from  these  assumptions,  especially  as 
they  impact  calculations  of  planned  confidence  intervals  and 
significance  statements. 

(7)  Discusses  how  the  independence  of  data  points  will  be 
pre"  ved  and  how  the  many  factors  which  typically  influence  the 
uti  y  or  character  of  a  data  set  will  be  controlled. 

(8)  Identifies  known  constraints  on  the  use  of  data  in 
support  of  the  evaluation  and  discusses  how  the  constraints  are 
to  be  handled.  (Examples  of  constraints  are  as  follows:  data 
from  a  model  which  does  not  play  relevant  hostile  or  friendly  air 
defense,  data  obtained  from  a  single  environment,  data  from 
immature  software,  logistics  data  limited  to  realistic 
maintenance  below  direct  support,  and  data  from  crews  which  have 
not  been  cross-trained.) 
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(9)  Includes  a  discussion  of  whether  the  constraints  will 
be  handled  jud^entally  or  through  the  use  of  formal  analysis 
(specify  technique) . 

(10)  Clarifies  the  extent  to  which  the  impact  of 
constraints  is  likely  to  be  remedied. 

c.  The  analyst  proposes  data  displays,  tables,  figures,  or 
other  forms  of  presentation  appropriate  for  displaying  data  in 
reports.  Tables  must  be  provided  for  cell  counts,  percentages, 
measures  of  central  tendency  or  variability,  and  for  ANOVA 
tables.  Group  MOE/MOP  by  like  presentations. 

3-104.  Analytic  summaries 

At  the  end  of  the  listing  of  MOE/MOP  for  each  criterion,  describe 
the  logical  process  which  the  analyst  intends  to  use  to  resolve 
the  criterion.  Group  the  MOE  and  MOP  by  like  analytic  designs 
and  concepts  and  describe  how  the  data  from  the  MOE  and  MOP 
sources  will  be  integrated  and  how  anticipated  constraints  on  the 
realism  or  completeness  of  the  data  will  be  treated.  At  the  end 
of  the  listing  of  criteria  for  each  issue,  describe  the  logical 
process  which  the  analyst  intends  to  use  to  resolve  the  issue. 
Group  analyses  by  like  analytic  designs  and  concepts  and  describe 
how  the  data  from  the  criterion  summaries  will  be  integrated  and 
how  anticipated  constraints  on  the  realism  or  completeness  of  the 
data  will  be  treated.  Describe  the  steps  that  will  be  used  to 
interpret  analyses.  Describe  how  and  where  modeling,  simulation, 
or  military  judgment  will  be  used.  Comparisons  are  made  to 
baseline,  to  criteria,  and  between  or  among  conditions.  Describe 
any  planned  statistical  procedures  or  analyses  that  were  not 
given  in  the  planning  for  the  individual  MOE/MOP  and  which  will 
be  used  to  combine  or  roll  up  two  or  more  measures  associated 
with  this  criterion.  Include  a  discussion  of  any  planned 
comparisons  or  statistical  testing.  (See  Section  X  for  details.) 

3-105.  DSM  . 

All  sources  of  data  to  be  used  in  an  evaluation  or  assessment 
must  be  described  in  the  TEP.  A  DSM  is  prepared  for  the  TEP. 

Each  MOE  or  MOP  must  be  addressed  by  at  least  a  primary  data 
source.  Any  test,  model,  simulation,  or  market  survey  or 
investigation  listed  as  a  primary  or  secondary  data  source  for 
any  MOP  must  be  described  in  chapter  2 .  Section  XI  provides 
details  on  the  development  of  a  DSM.  This  paragraph  is  only 
required  for  a  full  evaluation  TEP.  Abbreviated  evaluations  use 
only  data  obtained  from  test  described  in  chapter  3. 

3-106.  OT  approach  ,  w 

Use  this  paragraph  to  describe  an  OT  for  which  the  evaluator  has 
derived  the  test  approach  and  concept  and  whose  test  design  is 
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described  in  chapter  3  and  beyond.  Section  XII  provides  detailed 
instructions  for  this  derivation.  This  paragraph  is  only 
required  for  a  full  evaluation  TEP.  Abbreviated  evaluations 
derive  the  test  approach  in  chapter  3. 

a.  All  sources  of  data  to  be  used  in  an  evaluation  or 
assessment  must  be  described  in  the  TEP.  A  user  test  listed  in 
the  DSM  as  a  primary  or  secondary  data  source  for  any  MOP  must  be 
described  in  a  user  test  approach  paragraph.  The  user  test  to  be 
described  in  chapter  3  of  the  TEP  requires  a  complete  description 
of  the  evaluator's  test  approach  and  concept  in  this  paragraph. 

b.  This  paragraph  varies  in  scope  and  complexity  depending  on 
the  sophistication  of  the  anticipated  operational  test.  It 
provides  the  basis  for  chapter  3,  Test  Design  (where  details  of 
the  test  design  are  documented)  and  ultimately  for  the  detailed 
test  plan.  Elements  of  this  paragraph  include  test  scope, 
factors  and  conditions,  test  design  matrixes,  and  sample  sizes. 

3-107.  Test  scope 

The  scope  identifies  the  types  of  test  scenarios  and  test  events 
that  are  required  to  address  the  operational  test  MOPs  (OTMOPs) . 
The  level  of  detail  includes  the  approximate  length  of  various 
test  scenarios;  the  number  of  test  items  to  be  exercised  in  each, 
the  type  and  size  of  player  and  support  units,  and  the  test 
environment  (in  terms  of  maneuver  area,  terrain,  vegetation, 
threat,  and  security) .  The  degree  of  operational  realism 
required  in  the  test  must  be  described.  Operational  testing 
simulates  actual  combat.  Thus,  the  degree  of  operational  realism 
required  depends  on  the  types  of  scenarios,  types  of  test  events, 
and  combinations  of  test  conditions  required  to  answer  the 
OTMOPs.  (See  paragraph  3-92.) 

3-108.  Factors  and  conditions 

Factors  are  th  test  variables  identified  as  likely  to  affect 
test  event  ou  me.  Conditions  are  the  discrete  values  that 
factors  assuir  are  expected  to  assume.  (See  paragraph  3-94.) 

3-109.  Test  design  matrixes 

Once  the  required  types  of  test  events  and  scenarios  have  been 
identified  in  the  scope  and  appropriate  factors  and  conditions 
have  been  formulated,  the  evaluator  develops  matrix (ces)  needed 
for  grouping  combinations  of  test  conditions  into  "trials," 
"missions,"  or  "phases."  Matrixes  are  formed  by  crossing  factors 
and  conditions  with  one  another.  Large  matrixes  should  be 
reduced  to  several  smaller  ones.  Normally  the  analytic  designs 
described  above  will  reflect  these  matrixes.  The  evaluator 
determines  how  many  test  events  of  each  type  are  likely  to  occur 
in  each  type  trial  or  mission  and  how  many  trials  or  missions  are 
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possible  during  the  available  test  days  or  weeks.  (See  paragraph 
3-95.) 


The  evaluator  must  control  the  risk  associated  with  anticipated 
analyses.  He  identifies  the  number  of  valid  events  to  be 
conducted  under  specific  combinations  of  test  conditions,  m 
identifies  the  considerations  (such  as  order  and  pairing) 
respect  to  the  conduct  of  those  events.  These 
needed  to  support  anticipated  analyses.  Based  on  the  ^®®^ 
matrix (ces),  the  evaluator  estimates  the  aggregate  ®®™P^®  ®i?®® 
likely  to  occur  under  each  combination  of  factors  and  conditions 
for  tLts  of  various  lengths  and  for  alternative  f ovulations  of 
test  elements.  Often  the  aggregate  sample  s^e  ^  In 

the  time  period  and  number  of  systems  available  ^ 

this  case,  statistical  procedures  can  only  be  used  to  estimate 
the  risks  involved  with  alternative  allocations  of  rvouv®®* 

When  there  is  more  flexibility  in  terms  of  scenario  duMtion, 
number  of  trial  repetitions,  number  of  systems,  and  other 
resources  available  for  testing,  more  rigorous  sta^stical 
procedures  can  be  used  to  determine  sample  sizes  which  will 
reduce  the  risks  to  specified  levels.  Once  sample  siz^  are 
determined,  they  are  tabulated  as  matrixes  Rowing 
valid  test  events  to  be  conducted  under  various  combinaV^J^s  of 
test  conditions.  The  capability  of  exiting  or  PJ0P®®f^ 
instrumentation  to  collect  valid  data  in  a  cost  ef fe^ V® 
is  another  test  design  consideration.  (See  paragraph  3-96.) 

3-111.  DAG  requirements  . 

The  evaluator  identifies  whether  or  not  there  a^^® 
for  a  DAG.  This  paragraph  outlines  the  role  of  the  DAG 
validation  and  in  engineering  analysis  of  test  data.  Details  o 
the  DAG  procedures  are  covered  in  chapter  5. 

3-112.  Other  sources  of  data 

All  sources  of  data  to  be  used  in  an  evaluation  or  as^ssm^t 
must  be  described  in  the  TEP.  In  the  DSM  each  MOE^OP  ®ust  he 
addressed  by  at  least  a  primary  data  source.  Any  te^,  model, 
simulation,  market  survey,  market  investig^^”'  study, 
analysis  listed  as  a  primary  or  secondary 

MOE/MOP  must  be  described  in  this  paragraph.  User  ^®®^®  ^ ° ^ 
than  those  designed  in  this  TEP,  contractor  tests,  and  DT  will  be 
described  in  sufficient  detail  to  provide  unders^nding  of  the 
test.  Modeling,  simulation,  market  surveys  ®*}^ .  ' 

studies,  and  analyses  will  be  described  in  sufficient  detail  to 
permit  understanding  of  the  effort. 

3-113.  Data  base  structure 
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The  evaluator  describes  a  data  base  file  structure  that  will 
enable  him  to  analyze  test  data  in  a  thorough  and  timely  manner 
and  to  integrate  it  with  data  from  other  sources.  He  organizes 
known  data  elements  according  to  this  file  structure  and  provides 
definitions  of  data  requirements  where  appropriate.  Evaluator 
specification  of  data  base  structure  and  content  is  required  to 
ensure  that  data  base  descriptions  of  test  events  are  sufficient 
to  permit  flexible,  thorough,  and  timely  analyses.  (See  Section 
XIII  for  details.) 

a.  Specify  files  required  and  label  each  with  a  short 
description  of  the  type  of  data  to  be  stored  in  each  file. 
Indicates  for  each  file  whether  it  is  to  be  automated. 

b.  Describe  the  architecture  used  to  organize  the  data  base. 
The  architecture  typically  consists  of  either  a  network  or 
relational  design  for  data  storage.  The  network  approach  may  not 
be  appropriate  for  operational  evaluation  applications  because 
only  data  previously  related,  by  defined  associations,  can  be 
processed.  The  relational  approach  organizes  data  into  arrays  of 
rows  and  columns.  Only  the  types  of  data  need  to  be  specified; 
the  potential  relationships  between  data  are  implicitly 
represented.  This  approach  permits  the  rapid  generation  of  a  new 
file,  by  combining  across  two  or  more  types  of  data. 

c.  For  each  file,  list  known  data  elements  and  provide 
sufficient  definitions  to  preclude  ambiguity  and 
misunderstanding . 

3-114.  Evaluation  limitations 

This  paragraph  states  known  limitations  of  the  evaluation  and 
those  data  sources  (to  include  the  operational  test)  which 
support  the  evaluation.  These  limitations  are  to  be  presented 
and  discussed  in  terms  of  the  impact  of  the  limitations.  This 
will  include  an  explanation  of  why  the  limitations  exist,  which 
COIC  are  impacted,  and  what  can  be  e  to  minimize  their  impact. 


Section  XV 

Introduction  to  Test  Design 


3-115.  Test  design  phase 

The  test  design  phase  consists  of  the  completion  of  planning  for 
the  test  design  and  provides  the  source  material  for  the 
preparation  of  chapter  3  of  the  TEP  for  full  evaluate  systems  or 
the  preparation  of  chapters  2  and  3  of  the  TEP  for  all  other 
types  of  tests. 
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a.  Evaluation  planning  for  OTE  of  full  evaluate  systems  is 
performed  lAW  Section  IV  through  Section  XIV,  above.  These 
sections  provide  methodology  on  development  of  chapter  2 
(Evaluation  Plan)  by  the  lOE  in  the  full  evaluate  TEP. 

b.  Analytic  planning  for  OTE  of  abbreviated  evaluate  systems 
and  for  tests  of  CBTDEV  or  TNGDEV  products  is  abbreviated  into  an 
Analysis  Plan  and  is  performed  as  a  part  of  the  test  design 
process.  See  Section  XVI  on  the  development  of  the  POA  by  the 
tester.  Development  of  the  POA  permits  the  tester  to  write 
chapter  2  (Analysis  Plan)  lAH  Section  XIV,  above. 

c.  The  test  design  phase  is  distinctly  separated  from  the 
detailed  test  planning  phase  so  that  the  logical  design  can  be 
reviewed,  changed  if  necessary,  and  approved  before  heavy 
investments  in  time  and  resources  are  applied  to  implementation. 

3-116.  Test  design  requirements 

The  test  officer  must  be  constantly  aware  of  the  requirements  for 
the  specific  type  of  test  assigned.  He  must  ensure  appropriate 
requirements  for  the  specific  type  of  test  are  addressed.  The 
extent  of  preparation  required  for  the  TEP  is  dependent  upon  the 
level  of  evaluation  for  the  test. 

a.  If  the  system  is  a  full  evaluate  level  system,  chapters  1 
and  2  of  the  TEP  will  be  completed  and  provided  to  the  tester  by 
the  evaluator  based  on  the  methodology  established  in  Section  IV 
through  XIV,  above.  Specific  guidance  on  the  test  description, 
test  design  concept,  data  requirements,  and  data  base  structure 
will  have  been  provided  to  the  tester  in  chapter  2  of  the  TEP. 
This  guidance  will  assist  the  tester  in  developing  chapter  3  of 
the  TEP. 

b.  For  Abbreviated  Evaluate  (AE)  system  tests,  the  tester 
will  perform  many  of  the  evaluation  requirements  in  addition  to 
the  tester  requirements  and  will  complete  both  chapter  2  and 
chapter  3  of  the  TEP.  For  these  systems,  evaluator  guidance  is 
limited  to  providing  issues  and  criteria  for  the  test. 

c.  For  FDTE,  CEP,  and  CT  tests,  the  tester  will  have  to 
perform  all  of  the  evaluation  requirements.  The  evaluator  will 
normally  have  no  input  to  these  tests  unless  they  are  being  used 
in  lieu  of  EUTE,  LUT,  or  FOT.  The  tester  must  complete  both 
chapter  2  and  chapter  3  of  the  TEP.  For  these  systems,  guidance 
on  issues  and  criteria  for  the  test  is  provided  by  the  test 
requestor  (normally  the  CBTDEV  or  TNGDEV) . 

3-117.  Test  design  purpose 
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The  purpose  of  the  test  design  is  to  provide  the  logical 
foundation  for  the  test.  It  can  most  logically  be  developed  in 
the  sequence  outlined  in  this  chapter.  Through  a  process  of 
analyzing  each  issue,  the  needed  data  are  identified.  Methods 
for  turning  the  data  into  results  are  stated  in  detail.  The  list 
of  test  events  vhich  must  occur  is  derived  from  the  data  elements 
(DE) .  In  addition  to  this  logical  core,  the  design  can  contain 
methodology  and  procedures  for  test  execution  and  reduction, 
analysis,  and  reporting  of  test  data. 


Section  XVI 

Pattern  of  Analysis  (POA) 


3-118.  Requirements  for  a  POA 

A  POA  will  be  developed  for  all  tests.  For  full  evaluate 
systems,  the  POA  translates  the  evaluator  developed  OTMOP  into 
DR.  For  other  tests,  the  POA  translates  evaluator,  CBTDEV,  or 
TNGDEV  developed  OIC  into  tester  developed  OTMOP  and  DR. 

3-119.  Development  of  the  POA 

The  analysis  of  issues  is  the  detailed  refinement  of  test  issues 
into  questions,  ending  with  data  requirements.  It  should  be 
developed  as  the  first  step  in  a  formal  test  design  and  is 
required  for  all  tests.  It  basically  consists  of  establishing  a 
POA.  The  POA  is  a  tool  that  helps  the  test  officer  determine 
what  data  needs  to  be  collected  in  order  to  answer  the  issue. 

All  other  elements  of  the  design  are  dependent  upon  the  quality 
of  the  analysis.  This  section  provides  guidance  to  the  test 
officer  when  preparing  a  POA. 

3-120.  Definitions  for  the  POA 

a.  An  issue  is  answered  either  by  a  finding  or  an  assessment. 

b.  A  finding  is  a  result  which  is  limited  to  fact  and  is 
based  on  objective  and  subjective  data.  There  are  two  types  of 
findings. 

(1)  The  first  type  includes  results  of  computations  or 
other  mechanical  operations  performed  on  the  data.  Calculation 
of  summary  statistics  or  compilation  of  user  comments  would  be 
findings  of  this  type. 

(2)  The  second  type  consists  of  the  results  of  preset 
procedures  being  applied  to  the  data;  i.e.,  procedures  which  have 
been  preestablished  prior  to  starting  the  test.  An  example  is  a 
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preplanned  test  of  a  statistical  hypothesis  or  establishment  of 
some  minimum  acceptable  performance  score  for  a  test  item. 

c.  An  assessment  is  an  interpretation  of  objective  and 
subjective  data  and/or  findings  based  on  the  test  officer's 
military  judgment. 

(1)  The  difference  between  assessments  and  findings  of  the 
second  type  is  somewhat  confusing  at  first  sight.  ^  The 
difference  is  preset  procedures.  If  the  application  of  3udgment 
is  precisely  written  into  the  appropriate  plan  and  then  approve 
as  part  of  the  plan,  the  result  is  a  finding.  If,  however,  the 
judgment  is  applied  by  the  test  officer  after  the  test  begins, 
the  result  is  an  assessment. 

(2)  An  assessment  must  be  supported  by  all  available  , 

evidence.  The  believability  of  an  assessment  is  directly  related 
to  the  amount  and  quality  of  the  evidence  upon  which  it  is  based. 

3-121.  Rationale  and  method  of  analysis 

Although  test  issues  may  seem  detailed  from  the  MATDEV,  CBTDEV, ^ 
or  TNGDEV  points  of  view,  they  are  very  general  from  the  tester  s 
point  of  view.  Consider  the  issue,  "What  is  the  comparative 
effectiveness  of  weapons  systems  A  and  B?"  A  test  officer  could 
not  immediately  determine  from  the  issue  what  tasks  he  should 
have  the  weapons  perform,  what  data  he  should  collect  on  the 
performance,  or  how  data  should  be  analyzed  to  make  a  comparison. 
A  partial  answer  to  the  problem  is  to  divide  the  issue 
several  simpler  questions  which,  when  answered,  will  provide 
sufficient  information  to  fulfill  the  parent  issue.  Inherent  in 
this  division  process  is  the  knowledge  of  the  independent  and 
dependent  test  variables  that  were  developed  during  the 
preliminary  analysis  phase.  The  subdivision  process  continues 
through  as  many  levels  of  subquestions  as  are  necessary  to 
establish  DEs. 

3-122.  Methods  of  subdivision 

The  process  of  subdivision  is  an  extension  of  the ^process  of 
identification  and  categorization  of  variables  which  was  begun 
during  the  preliminary  analysis  and  planning  phase.  At  that 
time,  the  relationship  of  the  variables  was  tentative  and  not 
established  in  detail.  Development  of  the  analysis  of  issues 
makes  the  relationship  of  the  variables  rigorous  and  precise. 

The  only  requirement  in  each  subdivision  step  is  that  the 
subordinate  questions  be  sufficient,  as  a  group,  to  answer  the 
parent  question.  There  is  no  limit  to  the  number  of  ways  of 
subdividing  questions.  However,  the  two  ways  most  frequently 
used  are  division  by  situations  and  division  by  component 
attributes.  Other  specific  methods  may  be  appropriate  for 
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particular  tests.  These  methods  involve  division  by  MOP's  or 
division  by  qualitative  subquestions. 

3-123.  Subdivision  by  situations 

Division  by  situations  provides  separate  consideration  of  a 
question  or  issue  under  various  pertinent  sets  of  conditions. 
These  sets  of  conditions  correspond  to  identified  treatments 
(independent  variables).  For  instance,  the  issue  "How  well  do 
the  proposed  scout  platoons  perform  compared  to  the  standard 
scout  platoons?"  could  be  divided  into  three  subissues,  each 
directed  toward  one  of  the  major  scout  mission  types  (figure  3- 
25) .  Further  divisions  could  be  based  on  the  doctrinal 
subdivisions  of  each  mission  type,  time  of  day  or  night,  or  any 
other  situational  basis  the  test  officer  considers  appropriate 
for  focusing  his  analysis. 

(INSERT  FIGURE  3-25) 

3-124.  Subdivision  by  component  attributes 

Division  by  component  attributes  is  used  to  identify  qualities  or 
characteristics  of  the  item  or  concept  which  need  to  be  known  to 
answer  a  parent  question.  These  attributes  either  lead  to  or  are 
identical  to  the  pertinent  dependent  variables.  For  instance, 
the  question  "What  are  the  logistics  implications  of  weapons 
system  A?"  might  be  broken  into  subquestions  addressing  the 
attributes  of  supply,  maintenance,  and  transportation  as  shown  in 
figure  3-26. 

(INSERT  FIGURE  3-26) 

3-125.  Subdivision  by  MOP 

Whatever  the  method  of  subdivision,  the  ultimate  goal  is  to  link 
general  questions  to  simple  measurable  data  elements.  The  key  to 
establishing  this  link,  within  the  process  of  subdivision,  is  the 
identification  of  each  MOP.  MOPs  are  subquestions  in  the 
division  process  which  can  be  answered  in  numerical  terms  and 
which  represent  identified  dependent  variables  by  which  the  item 
or  concept  will  be  judged.  They  provide  the  bridge  between 
qualitative  subquestions  from  above  and  Des  below. 

a.  Consider  the  qualitative  subquestion  "Was  weapons  system  A 
maintainable?"  If  the  responsible  test  officer  feels  that 
further  division  in  qualitative  terms  is  unnecessary,  he 
identifies  the  MOP  which  become  the  next  lower  level  of 
subquestions.  Possible  MOPs  for  this  example  are  shown  in  figure 
3-27. 

(INSERT  FIGURE  3-27) 
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b.  Further  subdivision  of  the  MOP  is  not  usually  difficult. 
In  most  cases,  the  next  step  is  identification  of  summary  data, 
followed  by  identification  of  Des.  Completion  of  this  process 
for  the  above  example  is  shown  in  figures  3-28,  3-29,  and  3-30. 

(INSERT  FIGURE  3-28) 

(INSERT  FIGURE  3-29) 

(INSERT  FIGURE  3-30) 

3-126.  Subdivision  by  qualitative  subquestions 
Not  all  dependent  variables  can  be  expressed  in  the  MOP  format. 
Some  are  qualitative  in  nature.  For  example,  the  subquestion 
••Were  organizational  communications  personnel  able  to  erect  the 
antenna  without  dif f iculty?^'  might  be  subdivided  into  some 
subquestions  requiring  quantifiable  data  and  some  requiring  only 
comments  or  personnel  responses.  Such  a  breakdown  is  shown  in 
figure  3-31.  The  first  two  subquestions  are  MOPs.  The  third 
subquestion  calls  for  a  consolidation  of  qualitative  data. 
NONQUANTIFIABLE  DATA  IS  JUST  AS  IMPORTANT  TO  THE  TEST  AS 
NUMERICAL  DATA.  Both  varieties  should  be  exploited  to  the 
fullest. 


(INSERT  FIGURE  3-31) 

3-127.  Documentation  of  the  POA 

The  final  product  of  the  subdivision  process  is  the  POA  which 
contains  all  the  issues  and  their  subordinate  subdivisions.  The 
relationship  between  issues,  questions,  and  the  various  levels  of 
subquestions  is  indicated  by  the  numbering  system  used  in  the 
figures. 

a.  If  issue  3  of  any  test  had  three  questions,  they  would  be 
numbered  3.1,  3.2,  and  3.3.  Further  subdivisions  of  any  question 
are  numbered  by  adding  a  decimal  point  and  consecutive  integers 
to  the  number  of  the  parent  subquestion. 

b.  The  POA  can  be  set  up  in  two  different  formats.  For  the 
TEP,  the  format  will  have  to  be  as  shown  in  figure  3~32.  As  a 
vorking  document,  the  format  can  be  set  up  as  a  dendritic  tree— — 
the  branching  lines  forming  subdivisions  resembling  the  branches 
of  a  tree  (figure  3-33) .  In  either  format,  subquestions  are 
labeled  only  with  their  appropriate  number. 

(INSERT  FIGURE  3-32) 

(INSERT  FIGURE  3-33) 
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c.  The  only  exceptions  are  subquestions  which  are  also  Des. 
These  subquestions  are  labeled  with  the  letters  DE  in  addition  to 
the  number.  This  prefix  signifies  that  no  further  subdivision  of 
that  question  will  be  performed. 

d.  The  process  of  subdivision  contains  as  many  steps  as 
necessary.  There  is  no  limitation.  The  number  of  subdivisions 
will  almost  certainly  vary  from  one  issue  to  another.  The  only 
requirement  is  that  at  the  end  of  each  branch,  there  be  a  DE. 

e.  When  two  or  more  branches  are  subdivided  in  identical 
ways,  redundant  publication  is  unnecessary.  After  the  first 
branch  is  developed,  the  others  require  only  a  reference  to  the 
first. 

3-128.  Use  of  the  POA 

The  POA  is  the  backbone  of  the  test.  If  the  test  officer  has  any 
doubts  during  POA  development,  he  should  work  the  POA  with  the 
supporting  analyst  assigned  for  the  test.  Upon  completion,  the 
POA  should  be  closely  reviewed  within  the  test  organization  to 
ensure  that  it  is  complete,  unbiased,  and  logically  structured. 
Careful  review  and  revision  at  this  point  will  save  much  wasted 
effort  in  the  remainder  of  test  design. 

3-129.  Modifications  to  the  POA 

It  must  be  recognized  that  regardless  of  the  amount  of  effort  and 
review  that  a  POA  receives,  it  will  probably  require  some 
modification  as  the  planning  process  continues.  To  facilitate 
modification  and  review,  a  wallboard  format  for  the  POA  is 
generally  preferable  to  producing  and  reproducing  a  typed 
dendritic  outline.  The  primary  value  of  the  POA  lies  in  its 
value  as  a  planning  tool. 


Section  XVII 

Derivation  of  the  Test  .sign 


3-130.  Decomposition  oi  OIC 

OIC  are  stated  in  chapter  1  of  the  TEP.  Chapter  2  of  the  TEP 
decomposes  these  OIC  into  MOP.  Chapter  3  of  the  TEP  takes  those 
MOP  designated  as  OTMOP  (all  of  the  MOP  for  test  of  AE  systems, 
FDTE,  CEP,  and  CT)  and  derives  the  test  design. 

3-131.  Derivation  of  data  requirements  (DR) 

Associated  DR  and  resultant  test  factors  and  conditions  are 
specified  in  chapter  2  or  3  of  the  TEP,  as  appropriate.  For  full 
evaluate  systems,  the  evaluator  identifies  data  needed  to  support 
the  planned  evaluation  and  indicates  those  that  are  required  from 


3-60 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


test.  The  test  designer  includes  these  DR  and  derives  additional 
DR  needed  for  test  control,  diagnosis  of  problems,  interpretation 
of  the  data,  and  quality  assurance  (e.g.  the  tester  typically 
adds  the  DR  necessary  to  track  system  utilization  in  accordance 
with  the  OMS/MP) . 

3-132.  Initial  planning  for  data  collection 
Specific  instrumentation  is  identified  to  collect  the  data 
required,  including  relevant  constraints  on  its  use.  In  cases 
where  the  primary  means  of  data  collection  is  uncertain  of 
success,  a  backup  means  of  data  collection  is  included.  To 
ensure  that  a  proper  audit  trail  has  been  developed,  the  test 
designer  lists  each  DR,  the  instrumentation  or  other  means  of 
data  collection,  and  the  degree  of  data  accuracy  and  precision 
required. 

3-133.  Definition  of  the  test  structure 

The  structure  of  an  operational  test  is  driven  by  sample  size 
requirements  and  resource  limitations  and  differs  as  a  function 
of  its  complexity.  Tests  can  be  made  up  of  a  training  phase, 
pilot  test,  several  scenario-driven  test  phases,  and  special 
exercises  or  excursions.  The  test  designer  defines  the  test 
structure  and  for  each  element,  specifies  what  is  to  be  included 
and  how  it  is  to  be  accomplished.  Based  on  the  OT  concept,  the 
test  designer  describes  the  phases,  scenarios,  trials,  and  test 
events  to  be  executed  and  defines  objectives  for  each. 

3-134.  Tradeoffs  in  test  design 

The  trade-off  between  practicalities  of  test  conduct  and  analytic 
interpretability  of  results  is  a  primary  conce’-n  as  the  tester 
organizes  test  events  into  test  trials  and  test  trials  into 
scenarios  and  phases.  Fewer  data  points  collected  under  the 
right  circumstances  can  provide  more  information  than  many  data 
points  collected  under  less  desirable  conditions. 

3-135.  Design  for  factors  and  conditions 

A  key  consideration  is  frequency  of  required  systematic  changes 
in  test  conditions.  Some  systematically  varied  test  factors  are 
easier  to  change  in  the  field  than  others. 

a.  Fewer  changes  in  systematically  controlled  factors  enhance 
efficiency  of  test  conduct.  Fixing  a  combination  of  controlled 
test  conditions  and  conducting  a  number  of  trials  before  changing 
the  combination  of  controlled  test  conditions  is  the  easiest  way 
to  test.  Such  an  approach  has  drawbacks  both  for  operational 
realism  (especially  player  uncertainty)  and  for  analytic 
interpretability  (confounded  factors  and  spurious  or  inflated 
claims  of  statistical  significance) . 
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b.  Frequently  changes  in  systematically  controlled  factors 
enhance  analytic  interpretability.  If  every  combination  of  test 
conditions  occurs  in  every  trial  or  in  trials  conducted  close 
together,  analysis  is  easier.  Such  an  approach  is  seldom 
operationally  realistic  or  feasible. 

c.  The  tester  compromises  between  efficiency  of  test  conduct 
and  analytic  interpretability  by  considering  the  ease  with  which 
test  conditions  can  be  manipulated.  Those  conditions  that  are 
relatively  easy  to  change  are  varied  for  trial  to  trial  (or  even 
within  trials) .  Test  conditions  that  are  hard  to  change  are 
varied  over  longer  periods  (e.g.  daily,  weekly).  The  tester 
builds  a  test  design  which  is  both  feasible  to  conduct  and  likely 
to  yield  analytically  interpretable  results. 

d.  Statistically,  such  an  approach  yields  a  "blocked”  design. 
Through  knowledgeable  application  of  this  statistical  design 
approach,  the  tester  can  ensure  systematic  manipulation  of  many 
factors  in  an  operationally  realistic  manner  and  encourage  an 
appropriate  degree  of  player  uncertainty  over  extended  test 
periods. 


Section  XVIII 

Analysis  of  Test  Requirements 


3-136.  Tester  role 

For  full  evaluate  systems,  the  analysis  of  the  test  requirements 
is  performed  by  the  evaluator.  For  tests  of  AE  systems,  FDTE, 
CEP,  and  CT,  the  tester  must  decompose  issues  into  OTMOP  and  DR. 

3-137.  Addressing  issues 

The  analysis  process  begins  with  examining  the  testable  QIC  in 
c*  oter  1  of  the  TEP  and  is  followed  by  determining  how  the 
;  I as  will  be  addressed  in  terms  of  test  variables,  method  of 

€  uation,  test  length,  and  sample  size.  This  process  includes 
ic  itification  of  the  test  purpose,  scope,  and  tactical  context 
for  the  test.  This  process  provides  a  general  level  of  test 
requirements . 

3-138.  The  test  purpose 

The  test  purpose  statement  provides  the  test  officer  with  the 
reason  for  the  test.  It  must  answer  two  questions:  Why  is  the 
test  required  and  what  will  be  done  with  the  test  results? 
Answering  these  questions  requires  an  understanding  of  the 
proponent's  view  of  the  test.  The  initial  research  should 
provide  the  answers.  Some  example  purpose  statements  are  shown 
in  figure  3-34.  Notice  that  in  every  case,  exclusion  of  the  last 
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sentence  (disposition  of  results)  would  make  the  purpose 
statement  incomplete. 


(INSERT  FIGURE  3-34) 

3-139.  Scope  and  tactical  context 

The  scope  and  tactical  context  provide  the  general  conditions 
under  which  the  test  will  be  performed. 

a.  The  scope  is  a  narrative  of  the  location,  duration,  and 
overall  timeframe  of  the  test  and  a  summary  of  the  testing  to  be 
accomplished.  The  particular  information  included  varies  from 
test  to  test.  Subjects  may  include  quantity  and  type  of  forces 
involved,  requirements  for  control,  requirements  for 
instrumentation,  desired  levels  of  realism,  need  for  live  fire, 
division  of  the  test  into  phases,  nature  of  data  (objective, 
subjective,  MOPs,  etc.)  to  be  collected,  and  any  specific  test 
events  required.  Information  such  as  general  test  concepts  and 
design  based  on  employment  of  the  tested  system  or  concept, 
limitations  or  advantages  inherent  in  the  test  concept,  and  the 
comparison  of  items  with  descriptions  of  the  characteristics  to 
which  the  test  item  is  to  be  compared  may  be  cited. 

b.  The  tactical  context  is  a  description  of  the  military 
environment  in  which  the  item  or  concept  will  be  tested.  As  in 
the  scope,  the  information  included  varies  from  test  to  test. 
Subjects  may  include  the  assumed  level  of  conflict  intensity 
(e,g.,  mid-intensity),  description  of  the  threat,  missions  to  be 
performed  by  the  test  unit,  summary  of  the  scenario  (including 
reference  to  the  applicable  TRADOC  standard  scenario) ,  nature  of 
terrain  and  environment,  and  departures  from  doctrine.  Realistic 
battlefield  environmental  conditions  will  be  described. 

3-140.  QIC  and  OTMOP 

QIC  is  a  generic  term  which  is  used  to  include  issues  to  be 
addressed  in  user  testing.  FDTE,  CEP,  and  CTs  will  normally  use 
the  QIC  for  the  issues  and  criteria  to  be  addressed  in  test. 
Materiel  and  IMA  program  tests  will  normally  use  the  terms  QIC, 
COIC,  and  AOIC.  Operational  issues  are  questions  with  associated 
criteria  which  are  addressed  by  the  evaluator.  The  OIC  include 
the  COIC  developed  by  the  CBTDEV  and  the  AOIC  developed  by  the 
evaluator  to  complement  and  supplement  the  COIC.  OIC  cover  the 
aspects  of  a  system  which  are  applicable  to  the  OTE  of  that 
system.  See  Section  IV  for  a  detailed  discussion  of  OIC.  The 
approved  TEP  will  normally  identify  exactly  which  issues  (OIC  or 
COIC/AOIC)  should  be  addressed  by  OT,  by  DT,  and  by  other  means, 
such  as  modeling  and  simulation  or  market  surveys.  The  TEP 
provides  the  conditions,  range  of  conditions,  event  matrixes,  and 
data  requirements  for  a  user  test  of  the  evaluation  issues.  The 
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tester  may  determine  that  there  are  limitations  which  preclude, 
completely  or  in  part,  the  address  of  a  particular  issue  in  this 
user  test.  The  evaluator  is  informed,  and  the  limitation  is 
fully  explained  in  the  TEP. 


Section  XIX  ,  . 

Identification  of  Test  Variables,  Factors  and.  Conditions 


3-141.  Variables 

A  variable  is  some  measurable  or  describable  aspect  or  factor  of 
a  system  and/or  the  system  environment  which  is  subject  to 
change.  It  is  a  feature  of  the  test  item  or  concept  or  a  feature 
of  the  environment  which  can  be  observed  and  described. 

Variables  are  the  building  blocks  from  which  the  test  is 
designed.  For  this  reason,  identification  and  classification  of 
test  variables  are  critical.  There  are  three  types  of  test 
variables:  independent,  dependent,  and  extraneous. 

3-142.  Independent  variable 

An  independent  variable  is  a  factor  whose  behavior  is  not 
dependent  on  other  variables.  An  independent  variable  is 
normally  set  and  held  at  one  or  more  levels  during  the  test.  It 
is  intentionally  and  systematically  varied  or  allowed  to  vary  so 
the  effects  of  such  changes  on  the  dependent  variables  can  be 
observed.  For  example,  in  an  operational  test  of  a  machine  gun, 
the  independent  variables  could  include  range  of  engagement, 
target  type,  type  of  sight,  and  light  conditions. 

a.  The  number  of  levels  ^f  each  variable  is  the  number  of 
values  the  tester  plans  to  allow  the  independent  variable  to  have 
during  the  test.  In  the  example  given,  the  tester  might 
establish  the  range  of  engagement  at  three  levels — 200,  600,  and 
1,000  meters.  Similarly,  he  might  establish  the  light  c  nditions 
at  two  levels--day light  and  darkness. 

b.  An  independent  variable  is  systematically  varied  tan  a 
specific  plan  for  values  the  variables  will  assume  is  madts  a  part 
of  the  variable  treatment  plan. 

c.  An  independent  variable  is  tactically  varied  when  the 
variable  is  allowed  to  assume  any  value  which  results  from 
tactical  conditions  in  a  field  situation. 

3-143.  Dependent  variable 

A  dependent  variable  is  a  factor  whose  behavior  is  a  function  of 
one  or  more  identifiable  independent  variables.  The  tester 
observes  the  dependent  variable  to  see  the  effects  of  varying  the 
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independent  variables.  Changes  in  the  dependent  variables  should 
result  from  changes  in  the  independent  variables.  In  the  example 
given  above,  the  dependent  variables  could  include  percentage  of 
hits  and  average  miss  distance.  Dependent  variables  are 
sometimes  referred  to  as  MOP  and  are  often  specified  in  the  TEP 
as  part  of  the  criteria. 

3-144.  Extraneous  variable 

An  extraneous  variable  is  a  variable  that  is  not  selected  or 
cannot  be  selected  for  treatment  as  an  independent  variable  but 
is  expected  to  produce  some  effect  on  the  dependent  variables. 

One  of  the  primary  considerations  in  designing  a  test  is  the 
desire  to  minimize  and/or  document  the  effects  caused  by 
extraneous  variables. 

a.  Some  extraneous  variables  are  called  random  error 
variables  because  they  vary  in  a  seemingly  random  manner  and  can 
cause  the  value  of  the  dependent  variables  to  either  increase  or 
decrease.  Weather  is  an  example  of  an  extraneous  variable  which 
produces  random  error.  Increasing  the  length  or  number  of 
iterations  of  a  test  tends  to  average  out  or  dilute  the  effects 
caused  by  random  errors. 

b.  Other  extraneous  variables  are  sometimes  called  constant 
error  variables  because  they  affect  the  dependent  variables  in  a 
constant  direction  and  cause  a  bias  in  the  results.  Differences 
in  leadership  or  training  between  units  used  to  test  different 
procedures  for  accomplishing  the  same  tasks  may 'introduce  such  a 
bias  or  constant  error.  Test  design  techniques  such  as  rotating 
units  and/or  operators  through  both  procedures  being  compared  may 
reduce  the  adverse  impact  of  constant  error  extraneous  variables. 

c.  Thorough  documentation  of  the  conditions  existing  during  a 
test  is  necessary  to  enable  the  analyst  or  the  user  of  the  test 
results  to  gauge  the  effects  of  extraneous  variables. 

3-145.  Variable  identification 

There  is  practically  no  limit  to  the  number  of  variables 
associated  with  a  test.  The  test  designer's  goal  is  to  find  the 
major  variables  which  may  impact  significantly  on  the  test  data. 
This  process  is  more  of  an  art  than  a  science,  but  the  following 
guidelines  may  help: 

a.  Visualization  consists  of  visualizing  the  item  '-r  system 
performing  its  mission  in  an  operational  environment.  During 
this  step,  the  test  officer  should  record  every  variable  he 
identifies.  This  is  often  a  good  time  to  enlist  the  aid  of 
personnel  of  varied  backgrounds.  Often  important  vari:2bles  which 
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are  not  indicated  from  other  sources  will  be  identified  based  on 
experience. 

b.  Analysis  of  test  issues  is  one  of  the  richest  sources  of 
significant  variables  and  the  primary  constraint  in  limiting  the 
number  of  variables  to  be  examined.  The  identified  courses  of 
action  imply  at  least  a  part  of  the  list  of  treatments.  For 
instance,  in  an  OT  the  decision  problem  might  involve  a  choice 
between  two  developmental  machine  guns.  This  would  make  the  type 
of  weapon  an  independent  variable. 

3-146.  Variable  classification  ^ 

Once  identified,  the  variables  should  be  recorded  and  classified 
similar  to  that  shown  in  figure  3-35.  This  classification  is 
tentative  and  may  change  as  planning  continues.  It  must  be 
that  such  variables  as  MOS  of  gunners  and  level  of  training  could 
either  be  held  constant  or  could  be  systematically  varied.  In 
the  latter  case  they  would  be  classified  as  independent 
variables. 

(INSERT  FIGURE  3-35) 

3-147.  Treatment  of  test  variables 

a.  After  the  variables  are  identified  and  classified,  it  is 
necessary  to  decide  how  the  independent  and  extraneous  variables 
will  be  handled.  These  decisions  will  form  the  basis  of  the  test 
design. 


b.  The  tester  must  specify  exactly  how  each  independent 
variable  will  be  allowed  to  vary  during  the  test.  For  example, 
if  data  is  to  be  collected  both  during  daylight  and  darkness,  the 
independent  variable  "light  condition"  is  said  to  be  tested  at 
the  levels  of  "daylight  and  darkness." 

c.  The  independent  variable*^  lould  be  varied  by  a  limited 

number  of  discrete  increments  to  re  that  data  can  be 

collected  under  comparable  condit,  3.  It  may  be  impossible  to 
make  any  valid  statements  concerning  the  percentage  of  hits  of 
machine  gun  A  at  800  meters  versus  the  percentage  of  hits  of 
machine  gun  B  at  900  meters. 

d.  For  example,  the  independent  variables  in  figure  3-35 

could  be  varied  as  shown  in  figure  3—36.  figure  3— 36^there 

are  48  distinct  combinations  of  the  five  independent  variables 
and  each  combination  will  generate  different  data  for  the 
dependent  variables. 


(INSERT  FIGURE  3-36) 
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e.  The  addition  of  another  independent  variable  (two  types 
of  ammunition)  would  increase  the  number  of  combinations  to  96 
(if  the  dependent  variables  are  to  be  measured  under  all  possible 
combinations) .  For  this  reason,  caution  must  be  exercised  at 
this  point  to  keep  the  number  of  combinations  and  therefore  the 
size  of  the  test  to  a  manageable  level. 

f.  The  tester  must  attempt  to  control  the  effect  of 
extraneous  variables.  In  some  cases  this  can  be  done  by  holding 
the  extraneous  variables  constant.  For  example,  in  figure  3-19, 
the  gunner  MOS  and  the  level  of  training  can  be  held  constant  by 
selection  of  test  participants.  Weather  can  be  controlled  to 
some  extent  by  the  selection  of  test  location  and  season. 

3-148.  Basic  test  designs 

The  overall  test  design  must  provide  the  tester  a  framework  that 
will  allow  for  the  logical  and  controlled  collection  of  data. 
Whenever  possible,  the  test  should  be  structured  to  allow  for 
evaluation  by  comparison  of  data.  There  are  three  basic  test 
designs  that  can  be  used  in  most  user  testing;  new  system  versus 
replaced  system,  with  versus  without,  and  new  system  versus 
predetermined  standard. 

a.  When  the  function  addressed  is  currently  being  performed 
by  a  standard  inventory  item,  the  comparison  may  be  between 
measured  performance  of  the  tested  system  and  the  performance  of 
the  standard  system  under  the  same  circumstances. 

b.  The  comparison  is  between  the  measured  performance  of  an 
organization  in  accomplishing  the  given  function  with  the  tested 
system  versus  without  the  tested  system. 

c.  The  comparison  is  between  measured  performance  of  the 
tested  system  against  a  set  of  predetermined  standards. 


Section  XX 

Test  Length  and  Sample  Size 


3-149.  Sample  size  requirements 

In  user  testing  the  determination  of  the  time  and  sample  size 
required  to  collect  the  appropriate  quantity  of  data  is  usually  a 
difficult  process  requiring  advance  planning.  For  example,  how 
many  operators,  machine  guns,  and  rounds  are  required  to  provide 
a  valid  estimate  of  hit  performance?  These  decisions  must  be 
made  based  on  the  population,  anticipated  dispersion  of  the  data, 
and  required  level  of  confidence.  Then,  based  on  combinations  of 
test  conditions,  the  required  test  time  can  be  estimated.  In 
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many  instances,  the  test  will  be  constrained  by  resource  and  time 
limitations.  Analysis  personnel  must  resolve  differences  between 
required  and  available  resources  and  time.  Paragraph  3-96,  above 
provides  a  detailed  discussion  of  sample  sizes.  There  is  some 
guidance  on  determining  sample  size  requirements  for  RAM  in  Mil- 
Std  7 8 ID  and  DOD  3235. 1-H,  and  in  Part  One. 

3-150.  Bases  of  sample  size 

In  general,  the  test  duration  and  sample  size  is  based  on  the 
minimum  amount  of  testing  required  to  reach  definitive 
conclusions  concerning  the  test  item  performance  as  needed  for 
evaluation  of  the  issues  and  criteria.  The  determination  of 
sample  size  and  duration  of  a  test  is  based  on  statistical 
procedures,  military  judgment,  or  a  combination  of  both, 
depending  on  the  item  under  test.  Likewise,  the  analysis  of  test 
results  may  be  statistical  or  subjective. 

3-151.  Statistical  sample  size  procedures 

For  those  occasions  where  statistical  procedures  are  appropriate, 
various  probabilistic  tables  and  curves  are  used  to  determine  the 
minimum  test  required  to  ensure,  at  a  desired  confidence  level, 
that  a  bad  item  will  not  be  accepted  by  the  appropriate  decision 
makers  even  though  it  seems  to  meet  performance  criteria  and  that 
a  good  item  is  not  rejected  by  the  appropriate  decision  makers 
even  though  it  does  not  seem  to  meet  the  criteria.  While  the 
procedures  outlined  in  AR  702-3  apply  specifically  to  R7^ 
testing,  similar  procedures  will  be  used  in  the  selection  of 
sample  sizes  to  test  other  criteria,  such  as  accuracy  of  location 
and  probability  of  detection.  Statistical  procedures,  for 
example,  are  almost  always  used  when  dealing  with  reliability 
criteria.  Reliability  requirements  should  be  stated  in  terms  of 
the  lower  test  limit  and  the  upper  test  limit. 

3-152.  Derivation  of  test  limits 

For  those  occasions  where  the  lower  test  limit  and  upper  test 
limit  are  nc  provided  in  the  requirements  document  or  the 
development  an,  the  test  directorate  should  request  that  the 
CBTDEV,  aft<  caching  an  agreement  with  the  MATDEV,  provide  the 
required  val  a.  If  the  directorate  cannot  get  the  required 
values  in  time,  the  test  officer  and  analyst  should  select  the 
values  based  upon  subjective  analysis.  These  values  will  be 
based  on  the  following  considerations  on  a  case-by-case  basis: 

a.  The  essential  and  desired  criteria  in  the  requirements 
document . 

b.  Military  experience  and  judgment. 

c.  Previous  test  results  of  the  item  or  similar  items. 
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d.  Resources  required  to  test  given  values  and  the  desired 
discrimination  between  "good”  items  and  "bad”  items. 

e.  Performance  levels  of  items  which  it  will  replace  and  the 
importance  of  any  additional  capabilities  of  the  test  item. 

3-153.  Test  sample  adequacy 

Statements  concerning  the  adequacy  of  the  sample  size  or  test 
duration  should  be  included  in  the  scope  of  the  TEP. 

a.  This  paragraph  should  provide  a  statement  concerning  the 
test  criterion  (if  one  can  be  identified)  which  controls  the 
overall  sample  size  or  test  duration.  For  example:  ^The  sample 
size  of  1,000  rounds  is  required  to  test  the  compatibility  of  the 
item  with  the  M198  and  M109A4  howitzers  using  the  M3,  M4,  M119, 
and  M203  propelling  charges." 

b.  The  adequacy  of  sample  size  with  respect  to  other  criteria 
should  also  be  addressed.  For  example:  "The  sample  size  of 
1,000  rounds  will  provide  a  30-percent  risk  to  the  user  that  an 
item  with  a  true  reliability  which  does  not  meet  the  lower  test 
limit  will  be  accepted  as  meeting  the  requirement."  and,  "A  10- 
percent  risk  to  the  developer  that  an  item  with  a  true 
reliability  which  meets  the  specified  value  will  be  rejected  as 
not  meeting  the  requirement." 

c.  If  the  sample  size  is  inadequate  to  provide  an  acceptable 
risk  with  respect  to  any  criterion,  this  should  be  emphasized  in 
the  scope  paragraph.  For  example:  "The  sample  size  of  six 
missiles  is  not  adequate  with  respect  to  the  required  in-flight 
capability  of  the  missile.  The  six  missiles  provide  a  70-percent 
risk  to  the  user  that  an  item  with  a  true  reliability  which  does 
not  meet  the  lower  test  limit  will  be  accepted  as  meeting  the 
requirement  and  a  40— percent  risk  to  the  developer  that  an  item 
with  a  true  reliability  which  meets  the  specified  value  will  be 
rejected  as  not  meeting  the  requirement." 

3-154.  Use  of  military  judgment 

If  statistical  procedures  are  not  appropriate  for  the 
determination  of  sample  size  or  for  analysis  of  test  results, 
then  military  judgment  and  subjective  analysis  must  be  used. 


Section  XXI 

Level  of  Realism  Required 


3-155.  Realism  requirements 
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Simulation  of  operational  realism  is  an  essential  part  of  almost 
all  user  tests.  Test  concepts  and  plans  that  do  not  propose  to 
check  for  possible  degradation  of  system  performance  due  to 
realistic  conditions  of  employment  fail  to  address  a  critical 
decision  area  and  can  seriously  reduce  the  value  of  the  test 
results.  For  this  reason,  both  DOD  and  DA  have  placed  continued 
emphasis  on  evaluating  new  equipment  and  tactical  concepts  under 
the  most  realistic  battlefield  conditions  possible. 

3-156.  Levels  of  operational  realism 

The  requirements  for  design  of  a  test  to  meet  a  realistic  test 
environment  must  include  an  appropriate  level  of  operational 
realism.  The  conditions  for  test  environments  will  normally  fall 
into  one  of  three  categories  of  operational  realism;  maximum, 
limited,  and  minimal. 

a.  Methods  available  for  this  type  of  test  include  initial  and 
update  briefings  for  players,  aggressor  action,  simulation  of 
higher  and  adjacent  units  through  message  traffic  and  others,  and 
direct  intervention  by  controllers.  Direct  intervention  is 
always  a  final  resort. 

b.  Requirements  for  this  type  of  test  are  similar  to  full 
tactical  simulation.  However,  the  control  procedures  are  usually 
less  elaborate. 

c.  Although  little  realism  is  simulated  in  tests  of  this 
type,  there  is  a  necessity  for  frequent  checks  and  close 
supervision  to  be  sure  that  an  item  or  concept  is  properly 
employed  by  the  user.  For  these  tests,  this  section  describes 
the  frequency  of  checkup  inspections  and  the  indicators  to  be 
checked.  For  example,  for  user  evaluation  of  flight  suits,  the 
test  officer  must  check  to  ensure  that  the  suits  are  being  worn 
on  a  regular  basis. 

3-157.  Realistic  Battlefield  Environmental  Conditions  (RBECs) 
RBECs  are  those  natural  and  artificial  (tactical)  (friendly  or 
enemy  induced)  elements  employed  for  the  conduct  of  OT  to  achieve 
as  realistic  and  representative  a  battlefield  as  possible. 

a.  The  consideration  of  RBEC  must  occur  early  in  the  test 
planning  process.  It  is  the  responsibility  of  everyone  in  the 
testing  process  to  ensure  that  the  issue  of  operational 
performance  in  a  realistic  environment  is  thoroughly  addressed. 
This  requires  that  issues  requiring  evaluation  under  RBEC  be 
identified  at  the  beginning  of  the  test  planning  phase  and 
carried  through  the  OTP,  TEP,  DTP,  and  the  TER/OA.  This  is 
necessary  to  ensure  the  validity  of  the  test  results. 
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b.  If  factors  such  as  natural  obscuration,  weather,  and 
terrain  are  important  to  the  test,  they  can  be  addressed  to  some 
extent  by  specifying  the  test  location  and  date.  Other  factors 
are  influenced  by  the  intensity  of  the  conflict  (conventional, 
chemical,  nuclear,  etc.)  and  the  operational  capabilities 
(current  or  proposed)  of  the  forces  involved. 

c.  Decisions  that  will  limit  the  realism  of  the  test 
(essential  RBECs)  must  be  fully  documented  in  the  test 
limitations  portions  of  the  test  plans  and  reports.  Areas  which 
should  be  addressed  in  the  formulation  of  the  definition  of 
conditions  include  employment  of  opposing  forces  (OPFOR) , 
scenario,  terrain,  obscurants,  weather  and  illumination, 
electronic  warfare,  and  NBC. 

3-158.  Opposing  forces  (OPFOR)  play 

Simulation  of  a  realistic  OPFOR  is  an  essential  element  of  most 
tests.  Validity  of  test  data  and  analysis  depends  on  the  degree 
to  which  test  conditions  coincide  with  likely  combat  conditions. 
Part  One  discusses  threat  for  T&E  in  detail.  Guidance  for  OPFOR 
development  is  available  from  four  major  sources. 

a.  The  System  Threat  Assessment  Report  (STAR)  is  an 
integrated  assessment  of  projected  enemy  capabilities  which  could 
neutralize  or  degrade  a  specific  friendly  system  or  concept.  It 
serves  as  the  basic  threat  document. 

b.  Threat  test  support  package  test  setting  and  threat 
elements  are  the  most  useful  for  defining  the  expected 
battlefield  realism. 

c.  FM  30-102  and  FM  30-103  are  accurate  basic  documents  for 
OPFOR  establishment.  These  aggressor  manuals  describe  a 
hypothetical  but  realistic  OPFOR  including  tactics,  equipment, 
and  organization.  This  source  is  unclassified. 

d.  TRADOC  standard  scenarios  consist  of  various  scenarios 
which  depict  U.S.  forces  in  conflict  with  real  world  OPFOR.  The 
scenarios  are  expressions  of  TRADOC  and  FORSCOM  emphasis  on  OPFOR 
realism.  Each  scenario  was  developed  by  CACDA  and  the  service 
schools  to  provide  standardized  realism  in  training,  testing,  and 
evaluation.  This  source  is  classified. 

3-159.  OPFOR  in  TRADOC  standard  scenarios 

The  first  responsibility  is  to  be  familiar  with  the  standard 
scenarios  to  ensure  that  they  suit  the  test  situation.  The  scope 
of  the  OPFOR  was  considered  when  the  OTP  or  RS  was  formulated. 
Testing  should  generally  examine  worst-case  situations.  Unless 
the  test  issues  are  specifically  limited  to  a  particular  set  of 
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parameters,  the  worst-case 

rorsriaseVori^^fnrXure^^ai;many  conartions  o 

af possible.  Development  of  the  analysis  smaller  sized 

environments  to  te  simulated.  t  standard  scenarios. 

units  than  are  portrayed  in  detail  m  rn 

3-160.  Refining  TRADOC  standard  scenarios sequence  and 
When  refining,  the  appropriate  standa  d  t,  the  test 

a.  Adequacy  of  scenario  version  to  test  requirements. 

b  OPFOR  troop  list,  training,  weapons  and  equipment 
simSiat?o«,  end  ?actics  and  strategy. 

c.  Approval  and/or  certification  of  OPFOR  requirements  by 
appropriate  agencies. 


Section  XXII 
Test  Structure 

3-161.  Development  of  the  test  description  of  how 

The  test  structure  ^  results  in  the  general 

the  test  will  be  ®  ^  for  the  test  execution  and 

parameters,  phases,  a  control  activities,  data 

generates  requirements  for  tes  test  design 

management,  and  other  ^  usually  derived  from  a  four-step 

planning.  The  ®  ^  refining  and  analyzing  the  variables, 

process  which  consists  of  ref in  g  preliminary 

factors,  and  conditions  of^ions  determine  the 

planning  phase;  ^ i ons •  structv  g  subtests,  phases,  or 

required  set  of  test  conditions,  srru  ^  determining 

Events  based  on  the  exeSut  -n.  The  DR  developed 
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adjustment  must  be  made  oasea  up 

the  test  issues. 
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a.  As  previously  stated,  independent  variables  or  factors  may 
be  controlled  in  one  of  three  ways;  tactically  varied, 
systematically  varied,  or  held  constant.  Tactically  varied 
factors  are  preferred  because  they  develop  as  a  result  of 
tactical  operations  employed  in  the  test  and  enhance  test 
realism.  When  test  time  is  limited,  systematically  varied 
factors  are  used  to  permit  examination  of  all  required  factors  in 
sufficient  quantity  for  effective  analysis.  Factors  are  held 
constant  for  the  test  when  prior  knowledge  or  testing  indicates  a 
preference. 

b.  Uncontrolled  factors  should  be  held  to  a  minimum.  When 
critical  factors  for  a  system  are  identified,  the  most  repre¬ 
sentative  are  developed  for  such  factors  with  the  number  of 
conditions  held  to  a  minimum.  For  example,  the  terrain  factor 
for  testing  a  tank  system  may  have  swampy,  flat,  rolling,  and 
mountainous  conditions.  Since  flat  and  rolling  are  most 
representative  of  expected  operational  areas  of  employment, 
swampy  and  mountainous  conditions  would  be  excluded  unless 
essential  to  understanding  the  system's  effectiveness. 

c.  Ordinarily,  average  conditions  are  most  representative, 
but  extreme  test  conditions  could  be  more  appropriate  for  user 
testing.  For  example,  the  training  level  used  in  user  testing  to 
address  specific  issues  could  be  untrained  or  highly  proficient 
instead  of  average.  An  example  of  typical  factors  and  conditions 
is  shown  in  figure  3-37. 

(INSERT  FIGURE  3-37) 

3-163.  Combining  conditions 

The  selected  set  of  test  conditions  is  used  to  determine  what 
combinations  of  conditions  are  appropriate. 

a.  For  example,  a  hypothetical  system's  target  detection 
capability  could  be  influenced  by  three  training  level  conditions 
(untrained,  average,  and  highly  proficient) ,  three  weather 
conditions  (clear,  overcast,  and  precipitation) ,  and  two  terrain 
conditions  (flat  and  mountainous)  and  would  require  consideration 
of  18  possible  test  combinations  (3x3  x  2  =  18).  The  radio 
communications  capability  of  the  hypothetical  system  could 
require  consideration  of  training  and  terrain  conditions  (3  x  2  = 
6  combinations)  because  weather  conditions  have  little  effect. 

b.  A  suggested  technique  is  to  draw  a  matrix  listing  possible 
combinations  that  interact  and  influence  system  performance. 
Normally,  systematically  varied  controlled  factors  form  the  basis 
of  this  matrix. 
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3-164.  Structuring  subtests,  phases,  and  events 
Combinations  of  conditions  are  considered  for  selection  in  the 
least  number  of  groupings  which  will  allow  a  practical  and 
realistic  exercise. 

a.  Care  must  be  taken  to  avoid  groupings  of  incompatible 
conditions  (e.g.,  live  fire  versus  live  targets)  and  unrealistic 
conditions  (e.g.,  concentrated  armor  attack  under  nuclear 
attack) . 

b.  Each  grouping  becomes  one  test  subtest,  phase  or  event.- 
The  tester  identifies  the  types  of  trials  which  are  to  be 
conducted  and  the  conditions  under  which  they  must  occur.  He 
then  organizes  them  into  appropriate  groupings  identified  as 
phases  or  subtests. 

c.  Substantially  different  test  phases  generally  require 
separate  matrixes.  Test  event  matrixes  typically  require 
supplemental  notes,  annotations,  or  accompanying  text  to  clarify 
information.  In  particular,  such  supplemental  material 
identifies  the  manner  in  which  events  are  grouped  into  trials, 
missions,  and  phases  and  provides  guidance  concerning  sequencing 
or  prioritizing  the  conduct  of  test  trials  and  their  embedded 
test  events. 

d.  Done  properly,  implementation  of  this  process  by  the 
tester  ensures  appropriate  operational  realism,  prevents  players 
from  second  guessing  test  events,  and  minimizes  potential  bias  in 
the  data.  It  allows  flexibility  of  test  conduct  and  provides 
sufficient  test  control  to  meet  safety,  instrumentation,  and  test 
management  requirements.  An  example  of  a  test  phase  matrix  is 
shown  in  figure  3-38. 


HSERT  FIGURE  3-38) 

3-165.  Number  of  requ  i  trials 

The  number  of  trials  fo.;  an  event  or  sequence  of  events  is 
normally  dictated  by  statistical  criteria  required  to  support. the 
evaluation  procedures  and  analysis  stated  for  the  issue, 
criteria,  or  MOP.  The  required  sample  size  is  determined 
numerically  by  defining  statistical  parameters  and  formally 
calculating  the  sample  size.  Where  there  are  no  statistical 
criteria,  test  designers  must  determine  how  many  test  trials  are 
necessary  to  average  out  chance  differences  between  repetitions 
of  specific  events.  Essentially,  this  process  determines  how 
many  repetitions  are  required  to  provide  confidence  in  value 
derived  from  summing  trials.  A  trial  matrix  is  developed  for 
each  set  of  requirements  to  show  the  required  number  of 
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iterations  to  be  conducted  to  achieve  the  necessary  level  of  data 
collection.  An  example  event  matrix  is  shown  in  figure  3-39. 


(INSERT  FIGURE  3-39) 

3-166.  Use  of  test  structure  ^  ^  -4.^ 

When  completed,  the  test  structure  provides  the  test  officer  with 
the  general  plan  that  can  be  used  for  the  development  of  a  test 
schedule  and  execution  and  control  requirements  for  all  other 
aspects  of  the  test.  The  test  officer  uses  this  information  as 
the  foundation  of  chapter  3  of  the  TEP. 


Section  XXIII 
Test  Control 


3-167.  Test  control  planning 

This  section  describes  the  procedures,  documents,  and  functions 
that  must  be  developed  to  ensure  that  the  test  can  be  properly 
organized,  controlled,  and  executed  to  generate  the  required  test 
data.  The  basic  products  of  this  action  are  the  development  of 
the  TEP  Control  Concept  and  the  DTP  Control  Plan.  The  actions 
and/or  documents  described  in  this  section  must  be  performed  as 
developed  in  order  to  provide  detailed  instructions  and 
procedures  for  control  and  execution  of  the  test. 

3-168.  Program  of  events 

This  section  provides  both  an  outline  of  what  should  occur  during 
the  test  and  a  detailed  time  schedule  of  explicit  events.  The 
organization  and  level  of  detail  of  the  program  of  events  depend 
upon  the  level  of  simulated  realism  required. 

3-169.  Overall  test  scenario 

This  outline  divides  the  test  into  days,  mission  profiles, 
events,  subtests,  phases,  or  any  other  convenient  divisions.  A 
day  or  a  mission  may  be  repeated  a  number  of  times  to  develop 
sufficient  data  as  required  by  the  event  matrixes.  Any  such 
repetition  should  be  scheduled  to  agree  with  ordering  constraints 
Qj-  randomization  requirements  in  the  matrix.  This  outline 
reserves  periods  within  which  events  occur  and  serves  as  a  tool 
for  coordination.  General  procedures  are  developed  to  address 
delays  or  other  events  which  preclude  accumulation  of  scheduled 
data  due  to  weather,  instrumentation  failures,  system  failures, 
or  other  such  events. 

3-170.  Detailed  test  scenario 

The  detailed  test  scenario  consists  of  a  time  schedule  of  events, 
a  description  of  events,  and  associated  documents  for  event 


3-75 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


inputs.  Based  on  the  overall  test  scenario,  realistic 
operational  events  are  scheduled  into  appropriate  time  blocks 
using  enemy  threat  documentation,  U.S.  doctrine  and  tactics,  and 
test  constraints. 

a.  For  test  scenario  purposes,  an  event  is  something  that  can 
be  controlled  and  scheduled  to  occur  at  a  specified  time.  For 
example,  the  initial  conditions  (time,  location,  tactics,  force 
sizes)  for  an  opposing  force  ambush  of  a  friendly  infantry 
platoon  can  be  specified  in  advance,  so  such  an  ambush  can, 
therefore,  be  considered  an  event.  However,  the  resultant 
actions  following  the  ambush  are  generally  unpredictable,  so  they 
would  be  regarded  as  test  outcomes  rather  than  test  events. 

b.  Appropriate  documents  used  to  support  or  initiate  test 
scenario  events  may  be  operations  orders,  intelligence  reports, 
fragmentary  plans,  and  simulated  enemy  documents.  The  completed 
detailed  scenario  is  the  principal  control  mechanism  for  the 
test. 

c.  A  major  function  of  the  pilot  test  is  to  examine  the 
adequacy  of  this  mechanism  and  identify  changes  necessary  for  the 
actual  test.  Procedures  are  developed  to  monitor  scenario 
execution  during  the  actual  test  to  verify  that  programmed  events 
are  executed  in  accordance  with  the  detailed  scenario. 

3-171.  Full  tactical  simulation 

For  tests  which  require  simulation  of  a  tactical  environment  for 
extended  periods,  the  program  of  events  will  consist  of  a 
complete  scenario. 

a.  Essentially,  a  scenario  is  a  narrative  which  merges  the 
test  events  into  a  realistic  and  believable  sequence.  A  scenario 
describes  the  actions  of  all  player  and  OPFOR  units  and  includes 

?  1.1  information  which  will  be  presented  to  the  players.  This 
formation  includes  initial  and  update  situation  briefings, 
©rations  orders,  fragmentary  orders,  intelligence  summaries, 
ssages,  and  other  information  designed  to  evoke  player 
response . 

b.  Particular  attention  must  be  paid  to  the  actions  of  OPFOR. 
Their  operations  should  in  all  cases  be  consistent  with  the 
tactics  of  the  threat  force  being  considered. 

c.  All  scenarios  must  be  based  on  one  of  the  TRADOC  standard 
scenarios.  The  particular  standard  scenario  to  be  used  is  agreed 
upon  by  the  test  organization  and  the  proponent,  usually  during 
the  OTP  preparation  or  the  preliminary  test  planning  phase.  In 
preparing  the  scenario,  it  is  essential  to  specify  the  time  and 
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location  of  each  planned  event.  A  recommended  format  for  the 
scenario  is  shown  in  figure  3-40. 

(INSERT  FIGURE  3-40) 

3-172.  Limited  tactical  simulation 

In  tests  which  do  not  require  maximum  combat  realism,  the 
preparation  of  a  scenario  is  unnecessary.  It  is,  however, 
necessary  to  develop  a  detailed  description  of  the  events  that 
will  occur.  The  description  should  be  sufficiently  detailed  so 
that  the  events  can  be  properly  executed  without  additional 
information.  For  each  event  the  method,  time,  location, 
participants,  and  information  to  be  provided  must  be  specified. 

It  is  often  possible  to  group  similar  or  near  simultaneous  events 
into  subtests.  This  reduces  the  volume  of  necessary  description 
and  makes  conduct  of  the  test  more  efficient. 

3-173.  No  tactical  simulation 

For  tests  such  as  user  evaluations  and  some  customer  tests,  no 
simulated  realism  is  required.  For  such  tests,  the  event  summary 
can  be  augmented  as  necessary  to  provide  a  description  of  the 
conditions  or  situations  planned  or  expected  during  the  test. 
Examples  include  identification  of  test  units,  organization  of 
the  test  into  phases,  and  expected  frequency  of  key  events. 

3-174.  Need  for  control  procedures  •  v.  4- 

The  program  of  events  describes  in  detail  the  events  which  must 
occur.  The  control  procedure  supports  the  program  of  events  by 
specifying  the  system  which  will  cause  the  events  to  occur.  The 
content  of  the  control  procedures  depends  upon  the  level  of 
simulated  realism  required  in  the  test. 

3-175.  Control  procedures  for  full  tactical  simulations 
For  tests  of  full  tactical  simulation,  the  primary  control 
objective  is  to  induce  the  test  players  to  comply  with  the 
scenario.  In  support  of  this  objective,  the  control  procedure 
should  describe  the  control  organization  and  responsibilities, 
the  scenario  revision  and  documentation,  and  the  casualty 
assessment  requirements. 

3-176.  Control  organization  and  responsibilities 
The  control  organization  and  responsibilities  section  will 
describe  the  structure  of  the  control  group  down  to  individual 
level.  The  required  background  and  qualifications  for  each 
member  of  the  organization  will  be  specified.  The  mission  and 
responsibilities  of  each  subunit  will  be  described  in  detail.  No 
single  organization  is  applicable  to  all  tests.  Every  control 
organization  requires  a  centralized  headquarters  to  issue 
instructions,  reallocate  assets,  and  monitor  progress.  In 
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addition,  field  controllers  are  required  to  provide  information 
to  the  headquarters  and  to  intervene,  if  necessary.  These  field 
controllers  may  accompany  the  units  or  they  may  be  assigned  to 
locations  in  the  test  area.  Occasionally,  a  combination  of  unit 
controllers  and  area  controllers  is  necessary. 

3-177.  Scenario  revision  and  documentation 

Since  the  scenario  is  a  very  detailed  document,  it  is  to  be 

expected  that  changes  may  become  necessary  during  the  test. 

Major  changes  may  require  proponent  concurrence.  When 
information  from  field  controllers  indicates  that  a  change  has 
occurred  or  will  occur,  the  control  headquarters  should  be 
prepared  to  revise  the  scenario  in  order  to  ensure  the  occurrence 
of  required  events. 

a.  This  section  of  the  control  procedure  should  contain 
guidelines  for  revising  the  scenario  and  should  direct  procedures 
for  documenting  the  revisions.  Figure  3-41  is  an  example  of 
revision  guidelines. 


(INSERT  FIGURE  3-41) 

b.  The  best  procedures  for  documenting  a  revision  is  to  keep 
a  log  within  the  control  headquarters.  This  log  should  list  all 
test  events,  by  time  and  location,  as  well  as  directed  revisions. 
This  log  then  becomes  a  revised  scenario. 

3-178.  Casualty  assessment 

An  essential  element  of  full,  tactical  simulation  is  the  method 
used  for  assessing  casualties  during  engagements. 

a.  When  instrumentation  systems  are  used,  this  section  should 
describe  the  vehicle  and  weapon  simulations.  Guidance  on 
employment  of  instrumentation  systems  must  be  obtained  from  the 
agency  or  office  that  maintains  and  operates  that  system. 

b.  If  controllers  assess  casualties,  then  the  proc  ire  must 
be  specified  in  detail.  Guidance  on  assessment  procec  es  can  be 
found  in  plans  of  prior  tests  and  in  FM  105-5. 

c.  The  assessment  of  casualties  is  of  crucial  importance,  not 
only  because  it  directly  affects  test  data  but  also  because  it 
has  a  major  effect  on  troop  enthusiasm  and  morale. 

3-179.  Control  procedures  for  limited  tactical  simulations 
The  control  procedures  for  limited  simulation  tests  is  similar  to 
that  required  in  cases  of  full  tactical  simulation.  It  is 
usually,  however,  less  elaborate  since  the  control  function  is 
less  complicated.  Since  this  category  covers  a  wide  range  of 
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tests,  the  best  guidance  in  planning  the  control  procedures  will 
be  obtained  from  plans  of  similar  prior  tests.  As  a  minimum  the 
document  should  include  the  control  organization  and  functions,  a 
method  for  revising  the  program  of  events,  and  a  system  for 
documenting  the  revisions.  Casualty  assessment  may  be  required 
for  some  tests  of  this  type. 

3-180.  Control  procedures  with  no  tactical  simulation 

For  tests  with  no  simulation,  the  control  procedure  consists,  as 

a  minimum,  of -two  parts. 

a.  The  first  part  describes  the  supervision  and  monitoring 
that  the  test  officer  or  other  controllers  will  perform  to  assure 
that  the  test  item  or  concept  is  being  employed  properly.  The 
description  should  include  a  schedule  of  visits  or  inspections 
and  a  listing  of  points  to  be  checked.  Such  frequent  checks  are 
necessary  to  assure  that  misuse  or  misunderstanding  does  not  bias 
results  of  the  test. 

b.  The  second  part  should  outline  contingency  procedures  to 
be  employed  if  any  required  events  do  not  occur  in  the  course  of 
the  test  unit's  operations.  These  procedures  will  often  involve 
some  limited  simulation  testing.  Any  additional  information 
which  would  enhance  understanding  of  the  conduct  of  the  test 
should  be  included. 

3-181.  Pilot  test 

A  pilot  test  is  an  abbreviated  version  of  the  actual  test  and  is 
conducted  in  advance  to  detect  deficiencies  in  the  plan.  For 
this  reason  it  should,  as  a  minimum,  include  the  exercise  of  each 
type  of  required  event  and  make  use  of  each  data  collection,  data 
reduction  form,  and  mechanism  to  be  used  in  the  actual  test.  A 
complete  end-to-end  run  of  data  management  procedures  will  be 
conducted  as  a  part  of  the  pilot  test. 

a.  The  pilot  test  should  be  provided  for  in  the  control  plan 
with  sufficient  time  between  pilot  test  and  the  start  date  of  the 
actual  test  to  allow  for  reaction  and  correction  of  any 
deficiencies  encountered  in  control,  data  collection,  and  data 
reduction.  Tests  relying  heavily  on  instrumentation  may  require 
additional  time  after  the  pilot  test  for  the  correction  of 
problems. 

b.  Accomplishment  of  an  abbreviated  program  of  events  is 
usually  sufficient,  although  an  abbreviated  control  procedure  may 
also  be  required. 
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Section  XXIV 

Test  Data  Management 


3-182.  Data  management  planning 

This  paragraph  describes  the  procedures,  documents,  and  functions 
that  must  be  developed  to  ensure  the  collection  and  processing  of 
test  data.  Planning  data  management  functions  should  begin  early 
in  the  overall  process  and  continue  through  the  test  planning 
cycle.  Good  data  management  plans  are  essential  to  ensure  that 
all  the  resources  which  were  expended  in  the  planning  and 
execution  of  the  test  result  in  test  data  which  can  be  used  to 
address  the  issues  of  the  test.  The  basic  products  of  this 
function  are  the  Data  Management  Concept  in  the  TEP  and  the  Data 
Management  Plan  in  the  DTP.  These  include  data  collection,  data 
reduction,  quality  control,  and  written  procedures  for  reduction 
and  analysis.  Detailed  discussions  of  the  data  management 
function  are  provided  in  chapter  5. 

3-183.  Data  collection 

The  purpose  of  data  collection  is  to  assure  that  all  required 
data  generated  by  test  events  is  recorded  and  transmitted  to  the 
data  reduction  element  with  a  high  level  of  accuracy.  The 
functions  of  data  collection  include  gathering,  recording, 
checking  for  correctness  and  completeness,  and  safekeeping.  The 
plan  for  data  collection  is  based  upon  the  data  collection 
concepts  developed  during  the  test  design  phase.  The  plan  is 
divided  into  two  major  sections — the  first  section  deals  with  the 
data  collection  organization  and  its  functioning;  the  second 
section  concentrates  on  the  detailed  arrangements  for  recording 
data. 

3-184.  Data  collection  organization 

The  data  collection  organization  and  procedures  must  be  developed 
to  ensure  that  data  collectors  ad  instrumentation  will  be  in  the 
right  places  at  the  right  time  obtain  the  required  data. 

Plans  must  be  developed  to  desc  *  when,  where,  and  how  checks 

of  data  forms  or  instrumentatio  atputs  will  be  made  to  ensure 

that  data  are  being  collected  and  recorded  according  to  the 
definitions  and  formats  required. 

a.  The  data  collection  section  typically  consists  of  teams 
which  are  assigned  to  data  collection  stations.  These  stations 
may  be  moving  (e.g.,  with  a  tank  in  the  attack),  permanent  (e.g., 
at  the  same  location  for  the  whole  test),  or  temporary  (e.g.,  at 
a  given  location  during  only  one  part  of  the  test) . 

b.  Whereas  controllers  are  concerned  with  the  test  input  or 
stimulus,  data  collectors  are  concerned  with  the  test  conditions 
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and  the  output  or  response  of  the  test  item  and  player 
participants.  Some  data  collectors  may  be  tasked  to  operate 
automatic  or  semiautomatic  data  collection  equipment. 

c.  As  a  minimum,  the  plans  and  organization  must  cover  the 
following  items:  composition,  allocation  of  collection  means, 
gathering,  data  recovery,  and  coordination. 

(1)  A  chain  of  responsibility  is  established  and  mission 
statements  are  provided  as  required. 

(2)  Each  data  recording  agent  (human  or  instrument)  is 
assigned  to  a  player  unit  or  location,  and  the  data  to  be 
collected  are  identified.  These  arrangements  are  closely 
dependent  upon  the  control  plan.  Provisions  are  made  for  such 
items  as  relief  of  data  collectors  and  instrumentation  battery 
changes . 

(3)  Procedures  and  schedules  are  established  for  the 
periodic  pickup  and  control  of  manually  recorded  data  from 
collectors.  Prompt  consolidation  of  data  reduces  the  likelihood 
of  loss  and  allows  concurrent  monitoring  for  incompleteness  or 
error. 


(4)  Procedures  are  specified  for  attempting  recovery  of 
data  which  are  lost  or  erroneously  not  collected.  Such 
procedures  are  to  be  instituted  quickly  while  the  data  may  still 
be  available  or  can  most  easily  be  reconstructed. 

(5)  Plans  are  made  for  continuous  monitoring  of  the  test 
events.  Changes  in  the  sequence  or  timing  of  events  may  require 
reallocation  of  data  collection  means.  This  is  best  accomplished 
by  coordination  between  the  control  headquarters  and  the  data 
collection  headquarters. 

3-185.  Data  recording  procedures 

This  portion  of  the  plan  specifies  procedures  for  the  actual  act 
of  recording  the  generated  data.  The  method  of  recording  each 
data  element  is  specified  and  documented  in  the  TEP.  In  general, 
data  will  be  recorded  manually  by  the  data  collectors,  by 
instrumentation  alone,  a  combination  of  the  two  methods,  or  by 
the  players.  This  section  consists  of  the  data  forms, 
questionnaires,  and  instrumentation  codes  to  be  used  in  the  test. 
In  cases  where  instrumentation  is  used,  backup  data  forms  should 
be  considered  for  use  in  the  event  of  instrument  failure. 

3-186.  Data  reduction 

The  purpose  of  data  reduction  planning  is  to  establish  the 
organization  and  procedures  for  performing  preplanned  data  base 
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formulations  and  computations  on  the  test  data  in  accordance  with 
the  data  analysis  requirements.  The  contents  and  size  of  the 
data  reduction  plan  that  is  developed  are  highly  dependent  on  the 
magnitude  and  nature  of  the  reduction  effort.  This  plan  normally 
consists  of  the  data  reduction  organization,  reduction 
procedures,  data  base  development,  and  data  presentation  formats. 

a.  The  plan  for  the  data  reduction  organization  should 
identify  who  will  reduce  the  data  and  what  general  methods  will 
be  used.  The  reduction  element  is  divided  into  teams,  if 
necessary.  Also  included  in  the  plan  is  a  discussion  of  the 
functions  and  responsibilities  of  key  section  personnel  and 
personnel  rotation  schedules. 

b.  Data  processing  begins  when  complete,  properly  format! oo 
data  are  received  from  the  data  collection  section.  It  ends  when 
the  final  test  data  base  is  produced.  This  includes  entering 
manual  and  automated  data  into  an  automated  data  base  and 
performing  both  manual  and  automated  reduction  and  quality 
control  procedures,  as  well  as  constructing  and  maintaining  files 
to  provide  a  complete  test  data  audit  trail  from  collection 
(instrumentation  tapes,  video  tapes,  and  paper  files  of  original 
manual  data  forms,  data  collector  logs,  controller  logs, 
interviews,  etc.)  through  any  interim  reduction  process  to  the 
final  test  data  base. 

c.  Interim  reduction  consists  of  manual  or  automated 
procedures  which  transform  data  from  the  data  collection  means  to 
the  form  required  in  the  final  data  base.  Manual  reduction 
includes  processes  requiring  expert  analyses  and  judgment  such  as 
extracting  quantitative  data  from  video  tapes  and  examining  data 
from  multiple  sources  to  resolve  apparent  contradictions. 
Automation  is  used  whenever  it  can  sensibly  replace  manual 
reduction  procedures. 

d.  Re;  nsibilities  of  key  personnel  and  overall 

organizat:  1  structure  are  designed  to  accommodate  data  entry, 

data  reducv  .i  On  and  validation,  and  audit  trail  preservation.  The 
data  reduction  section  typically  requires  personnel  from  several 
agencies  or  contracting  organizations  at  many  different  levels 
possessing  skills  from  a  wide  variety  of  disciplines. 

Organization  of  the  section  should  allow  for  unusual  duty 
schedules  or  frequent  rotations  of  technical  personnel  to  permit 
timely  reduction  of  test  data  and  to  accommodate  personnel 
constraints  of  other  agencies. 

e.  Data  reduction  procedures  provide  detailed  instructions  to 
data  entry  and  reduction  on  the  actions  required  for  processing 
of  the  test  data.  Instructions  include  requirements  for  data 
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innut  individual  steps  for  reduction,  and  other  requirements. 

K  plaS  iay  be  Lcessary  when  reduction  is  to  ^  pertomed 

by  the  test  officer  or  by  a  small  group  of  ei^erience 
individuals  familiar  with  the  reduction  requirements.  It  is, 
however,  necessary  in  tests  where  the  reduction  organization  is 
very  large,  the  volume  of  data  is  very  large,  or 
sufficient  knowledge  or  expertise  to  execute  the  reduction  plan 
without  additional  detailed  instructions. 

f  Data  flow  descriptions,  both  within  the  data  management 

section  tnl  between  eleLnts  of  the  ‘s" Vi«?«nS!t?ar“ 
procedures  and  schedules  to  be  used  for  data 
particular  attention  is  paid  to  feedback  loops, 

data  manaaement  section  and  among  data  control,  collection,  and 
mlnagS  e?eLnts.  In  addition,  feedback  loops  ^om  emerging 
evaluation  and  statistical  analyses  are  generally  established. 

q.  Data  reduction  plans  describe  procedures  ®”®“^® 
inteqrity  of  the  test  data  base.  In  particular,  they  specify 
data  entry  and  updating  procedures  which  preclude  transcription 
^^orS  and  unautLrited  Lcess  and  provides  for  a  complete  audit 
trail  for  the  data. 

h.  Data  presentation  specifies  the  format  which  will  used 
to  present  summarized  test  results  in  the  final  ^  ^ 

evaluation  systems  this  is  usually  based  on  the 
techniques  proposed  by  the  evaluator  in  chapter  2  of  the  TEP. 

Lch  p?eplaLing  is  usually  necessary  for  ^®®^®. 
volume  or  extensive  variety  of  data.  For  such 

necessary  to  plan  presentation  formats  so  that  they  can  be  . 
p?odulS7du?ing  the  reduction  effort.  For  abbreviated  evaluation 
systems,  the  tester  develops  the  forms  data  - 

Reports  of  similar  past  tests  may  include  usable  methods  of 

presentation . 

The^quality ^control  procedures  should  describe  in  detail  the  data 
manaqement  procedures  which  ensure  that  the  correct  data 
collected  and  that  the  data  collected  are  correctly  processed. 

consist  primarily  of  checks  which  cannot  be  easily  automated 
^such  as  review  of  graphical  displays)  and  assessment  of  related 
data  from  several  sources  for  overall  consistency. 
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b.  Data  management  quality  control  procedures  provide 
mechanisms  for  adding  or  changing  data  requirements  and  for 
modifying  data  collection  or  reduction  procedures  based  on 
examination  of  emerging  test  data.  This  differs  from  the  quality 
control  procedures  in  data  collection  which  are  generally  limited 
to  assuring  that  data  are  completely  and  accurately  collected. 

c.  The  data  reduction  responsibility  is  to  ensure  that  data 
taken  as  a  whole  represent  what  actually  happened  during  test.  It 
stops  short  of  evaluative  analyses  of  test  data,  which  are 
primarily  concerned  with  the  interpretation  and  significance  of 
test  data  within  the  broader  framework  of  a  system  evaluation. 

3-188.  Data  authentication  group  (DAG) 

DAG  organization  and  procedures  are  documented  in  the  TEP.  The 
need  for  a  DAG  was  established  in  TEP  chapter  2.  A  discussion  of 
the  DAG  is  provided  in  chapter  5.  The  principal  product  of  a  DAG 
is  high-fidelity,  validated  data  bases  from  which  the  events  of 
the  test  can  be  revisited.  Resources  to  support  the  DAG  will  be 
identified  through  the  TSARC  process  on  the  OTP. 

3-189.  RAM  scoring  conference  procedures 

The  test  officer  should  develop  detailed  RAM  scoring  conference 
procedures,  in  conjunction  with  the  chairman  of  the  scoring 
conference,  for  inclusion  in  the  overall  data  management  plan. 
Formal  RAM  scoring  conferences  are  held  during  and  immediately 
after  an  operational  test.  The  objectives  of  the  scoring 
conferences  are  to  establish  a  common  test  data  base  and  to 
ensure  that  a  proper  and  consistent  categorization  (assigning 
classification  and  chargeability)  of  test  incidents  is  made  using 
the  approved  FD/SC.  Detailed  procedures  for  the  RAM  scoring 
conference  are  provided  in  Part  One  and  AR  702-3.. 


Section  XXV 
Test  Training 


3-190.  Training  of  personnel 

The  training  plan  is  an  essential  element  of  the  overall  test 
planning  process.  Requirements  for  training  are  developed  during 
the  evolution  of  the  control,  data  collection,  and  data  reduction 
plans.  For  each  category  of  personnel,  requirements  for  training 
are  identified.  A  program  for  eliminating  these  shortfalls  is 
then  developed.  Training  plans  include  designation  of  attendees 
and  instructors,  lesson  plans,  examinations,  performance  tests, 
schedules,  and  location  of  instruction.  Standard  Army  formats 
for  these  documents  should  be  used. 
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3-191.  Player  personnel  j  -4. 

Training  of  players  should  provide  individual,  section,  and  unit 
level  skills  necessary  to  operate  and  maintain  the  equipment  or 
perform  the  tactics  to  be  tested.  The  results  of  the  test  will 
be  sensitive  to  the  amount  and  quality  of  player  training.  In 
fact,  for  some  tests  the  amount  and  type  of  training  received  by 
various  groups  of  players  will  be  a  major  test  variable.  For  all 
tests,  the  actual  training  received  should  be  documented  and  will 
become  a  part  of  the  final  report.  The  tester  must  receive  the 
trainer's  Operational  Test  Readiness  Statement  (OTRS)  before 
proceeding  with  testing. 

3-192.  OPFOR  personnel  ,  ^  .  4.^ 

In  those  tests  where  OPFOR  is  included,  the  units  portraying  the 
enemy  should  be  trained  to  employ  the  tactics  of  the  threat 
force.  The  employment  of  U.S.  tactics  and  procedures  by  OPFOR 
may  invalidate  or  cast  doubts  on  results  of  such  a  test. 

Specified  tests  may  require  that  certification  of  threat  forces 
be  provided  and  a  high  level  of  training  required. 

3-193.  Test  controllers  .  .  4.  u 

Controller  training  requirements  describe  the  training  to  be 
given  to  the  control  section  personnel,  as  well  as  the  procedures 
to  be  used  to  determine  whether  training  was  effective.  At  a 
minimum,  controllers  are  to  be  completely  familiar  with  the 
control  plan,  the  data  management  and  collection  plans,  and 
system  safety  constraints.  Training  includes  test  issues,  scope, 
system  orientation,  safety  releases,  subtests  and  field 
exercises,  duty  schedule,  and  communications  procedures  or  radio 
nets.  Particular  attention  is  to  be  given  to  training 
controllers  not  to  cue  test  players  to  pending  events. 

Controllers  require  the  capability  to  make  sound,  independent 
decisions  on  control  matters  and  to  implement  emergency 
procedures  in  the  absence  of  communications.  Controllers  also 
require  the  ability  to  act  as  data  collectors  if  the  need  arises. 

3-194.  Data  collectors  .  4.  j  4. 

Data  collector  requirements  describe  training  to  be  given  to  data 
collectors  and  instrumentation  operators  as  well  as  procedures  to 
determine  whether  training  was  effective.  Data  collectors 
require  familiarity  with  the  control  plan  and  the  data  collection 
and  reduction  plans,  as  well  as  the  test  issues,  test  scope, 
subtests  and  field  exercises,  test  item  characteristics, 
instrumentation,  duty  schedules,  and  communications  procedures 
and  radio  nets.  Detailed  training  in  the  correct  use  of  data 
collection  forms  is  always  required.  Particular  emphasis  is 
given  to  the  need  for  accuracy  and  unbiased  objectivity. 

Selected  data  collectors  and  supervisors  may  require  detailed 

operations  or  maintenance  of  the  tested  system.  In 
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addition,  training  is  to  stress  that  data  collectors  are  not  to 
cue  test  players  to  pending  events.  Data  collectors  should  also 
be  trained  to  implement  emergency  procedures  if  necessary. 

3-195.  Data  reduction/entry  personnel 

Training  for  data  reduction/entry  personnel  should  provide 
familiarity  with  the  data  entry  requirements  and  data  reduction 
plan.  Formal  instruction  is  usually  necessary  and  training  time 
and  other  resources  must  be  planned  accordingly.  Practical 
exercises  with  dummy  data  will  provide  the  best  preparation. 


Section  XXVI 
Instrumentation 


3-196.  Instrumented  data  collection 

The  collection  of  test  data  through  the  use  of  automated 
instrumentation  systems  is  a  key  factor  in  the  majority  of  tests 
conducted.  In  addition,  instrumentation  systems,  to  include 
models  and  simulations,  are  often  employed  to  provide  realistic 
simulation  of  combat  environments  (weapons  simulator,  NBC^ 
simulants,  etc.)  and  to  generate  data  for  systems  to  use  in  lieu 
of  having  actual  forces  in  the  field  (combat  simulations  or 
stimulation  models) .  See  Part  Eight  for  a  detailed  discussion  of 
instrumentation . 

3-197.  Instrumentation  planning 

Instrumentation  planning  is  conducted  to  identify  those 
instrumentation  systems  that  are  required  to  collect  data  to 
address  the  test  issues  and/or  to  provide  the  necessary  degree  of 
combat  environment  realism  or  generation  of  cue/task  loading 
information.  The  test  officers  must  identify,  through  the 
overall  data  requirements  identification  test  control  procedures 
and  data  collection  reduction  planning,  the  detailed  requirements 
for  instrumentation  support.  See  chapter  f  or  details  of  the 
Instrumentation  Support  Plan  (ISP). 

3-198.  Instrumentation  sources 

Many  sources  exist  to  assist  the  test  officer  in  identification 
of  available  systems  and  the  system  capabilities.  Test  officers 
should  coordinate  with  the  instrumenters  as  early  in  the  test 
planning  phase  as  possible  for  identification  of  requirements. 
Instrumentation  engineers  will  assist  the  test  officer  in  the 
development  of  all  instrumentation  requirements  for  the  test. 


Section  XXVII 

Documentation  of  the  TEP  Test  Design  Plan  (TEP  Chapter  3) 
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3-199.  Test  design  planning  summary 

There  is  no  one  single  way  or  method  that  will  accommodate  the 
requirements  for  planning  for  all  of  the  types  and  requirements 
of  tests.  The  procedures  and  requirements  which  have  been 
discussed  serve  to  provide  the  test  officer  with  the  majority  of 
the  areas  which  should  be  considered  in  test  planning  for  any 
type  of  test.  Not  all  items  will  be  applicable  to  all  tests; 
however,  consideration  of  all  items  will  greatly  assist  the  test 
officer  in  overall  planning. 

3-200.  Documentation  of  test  design  planning. 

Results  of  test  design  planning  are  embodied  in  two  documents, 
the  TEP  and  the  DTP.  These  documents  are  the  methodology  by 
which  the  planning  requirements  are  documented  and  approved  for 
implementation . 

a.  In  the  TEP,  basic  test  design  is  discussed  in  TEP  chapter 
3.  This  design  planning  is  expanded  in  appendixes  to  the  TEP 
which  provide  the  concept  plans  for  test  scenarios,  test  threat, 
test  control,  test  data  management,  ITTS  support,  audiovisual 
support,  ADP  support,  test  training,  and  other  test  support. 

Each  of  these  topics  are  discussed  in  TEP  chapter  3 . 

b.  If  the  category  does  not  apply  (e.g.,  test  will  not  use 
ITTS) ,  this  fact  is  noted  in  chapter  3  and  no  appendix  is 
written.  If  the  requirement  is  very  simple  (e.g.,  threat  for 
tool  kit) ,  the  entire  threat  may  be  listed  in  chapter  3  and  no 
appendix  is  written. 

c.  The  concept  plans  in  the  appendixes  of  the  TEP  are 
expanded  into  detailed  plans  in  the  DTP.  Each  of  the  appendixes 
in  the  TEP  has  a  counterpart  in  the  DTP.  If  the  plan  is  simple, 
it  may  be  fully  developed  in  the  TEP.  If  the  plan  is  voluminous, 
it  should  be  summarized  as  a  concept  plan  in  the  TEP  and  fully 
developed  in  the  DTP. 

3-201.  Preparation  of  chapter  3  of  the  TEP 

This  chapter  is  prepared  by  the  tester  except  when  the  TEP  is 
used  to  plan  evaluation  for  a  milestone  that  will  not  be  preceded 
by  a  dedicated  phase  of  user  testing.  In  those  cases,  the 
evaluator  will  provide  a  brief  statement  to  that  effect  and  the 
remainder  of  the  chapter  is  not  required.  As  noted  above,  the 
POA  is  the  basis  for  derivation  of  this  chapter.  Using  the 
techniques  described  in  Sections  XVI  through  XXVI,  above,  the 
test  designer  develops  and  documents  the  test  design  as  shown  in 
figure  3-42. 


(INSERT  FIGURE  3-42) 
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3-202.  Test  description 

Describe  the  test  to  be  conducted  to  address  the  operational 
issues,  OTMOP,  and  data  requirements.  State  the  purpose  of  the 
test  and  where  the  results  will  be  used.  This  will  be  the  same 
as  the  test  purpose  contained  in  the  OTP.  Reference  authority  to 
conduct  the  test  to  include  the  DA  memorandum  approving  the 
current  FYTP  and  the  number  of  the  approved  OTP. 

3-203.  Test  overview 

Provide  an  overview  of  the  test  design.  The  purpose  of  the  test 
design  is  to  execute  the  concepts  described  in  chapter  2  of  the 
TEP. 


a.  Specify  the  test  phases  (e.g.,  pilot  test,  record  trials, 
live  firings,  FTXs,  CPXs,  demonstrations)  making  up  the  test  and 
summarize  the  test  factors  and  conditions  and  events  making  up 
each  phase.  Describe  the  use  of  any  baseline  in  the  testing. 
Briefly  describe  execution  procedures,  control  procedures,  and 
any  other  information  necessary  for  an  understanding  of  the 
matrix (ces)  that  are  applicable  to  all  issues.  Include 
information  such  as  limitations  or  advantages  inherent  in  the 
test  concept  and  the  comparison  items  with  descriptions  of  the 
appropriate  characteristics  to  which  the  test  item  is  to  be 
compared.  Include  provisions  to  ensure  that  test  events  adhere 
to  the  OMS/MP. 

b.  Describe  the  tactical  context  and  the  associated  scenario 
(for  example  Middle  East  or  European) ,  environment,  threat, 
tactics,  and  doctrine  to  be  used  in  each  test  phase.  As  an 
example,  the  maneuvers  of  the  friendly  forces,  threat  situation, 
terrain,  weather,  and  types  .of  events  to  be  represented  in  the 
test  can  be  included. 

c.  Review  and  refine  the  factors  and  conditions  in  chapter  2 

of  the  TEP.  As  necessary,  the  tester,  in  conjunction  with  the 
lOE,  refines  the  factors  and  conditions  by  adding  factors.  For  a 
given  factor,  the  tr  sr  refines  the  factors  by  changing  the 

number  of  conditions  redefining  the  conditions  or  changing  the 

type  of  control  (sys  matically  varied,  tactically  varied, 
uncontrolled,  or  held  constant) . 

3-204.  Conduct  of  test 

Describe  the  conduct  of  the  test  in  terms  of  its  tactical 
context,  events,  control,  phasing,  scheduling,  and  methodology. 

3-205.  Tactical  context 

Based  on  the  test  concept  in  TEP  chapter  2,  describe  the  tactical 
context  and  associated  scenarios,  environment,  threat,  tactics, 
and  doctrine  to  be  used  in  each  test  phase. 
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a.  The  scenario  description  consists  of  the  geographic 
portrayal  of  the  area,  forces,  and  events  of  a  hypothetical  armed 
conflict.  The  CBTDEV  provides  standard  scenarios  as  part  of  the 
D&OTSP  (Part  One) .  The  scenarios  provide  a  common  framework  of 
selected  situations  and  real  world  conditions  in  which  specified 
test  events  are  set  for  the  Middle  East,  Europe,  Alaska,  Korea, 
Persian  Gulf,  etc.  Expand  as  necessary  in  the  appendixes  to  the 
TEP  and  the  DTP. 

b.  Describe  the  use  of  test  site  terrain  for  each  phase, 
scenario,  mission,  or  trial.  Discuss  the  balance  between 
realistic  representation  of  the  operational  environment  and  the 
necessity  to  cause  the  required  conditions  to  occur  during 
programmed  test  events.  Describe  how  EW;  obscurants;  nuclear, 
biological,  and  chemical  (NBC);  level  of  conflict  intensity; 
mission  profiles;  and  environmental  factors  are  to  be  played. 

c.  Describe  how  the  threat  systems  and  tactics  will  be  played 
in  each  phase.  For  ACAT  I,  ACAT  II  and  DOT&E  oversight  systems, 
the  TTSP  and  the  STAR  describe  the  OPFOR  tactics  and  define  the 
threat  weapons  to  be  employed  and  the  size  and  type  of  the 
expected  force.  Describe  the  threat  in  sufficient  detail  to 
permit  realistic  testing.  Testing  must  include  an  accurate 
representation  of  the  threat  projected  to  exist  post-IOC.  Expand 
as  necessary  in  the  appendixes  to  the  TEP  and  the  DTP. 

d.  Describe  the  friendly  tactics  and  doctrine  to  be  played  in 
each  phase.  The  phases  are  designed  within  the  tactical  and 
doctrinal  framework  of  the  D&O  support  package.  Define  how  the 
framework  will  be  realized  on  the  ground  within  the  environment 
of  the  test.  Define  the  degree  of  test  player  free  play  to  be 
allowed  within  the  framework. 

e.  Include  data  on  test  player  forces  that  will  operate  the 
system  and  portray  the  supporting,  supported,  and  adjacent  forces 
in  play.  Identify  the  type  of  test  unit  or  organization  for  the 
test.  Discuss  how  the  unit  is  organized  and  any  significant 
requirements  of  the  unit.  Include  additional  information  such  as 
TOE  designation,  as  required. 

3-206.  Test  events 

Include  discussions  of  the  organization  and  overall  layout  of  the 
test  to  include  sequence  of  phases.  Flow  diagrams,  time  lines, 
and  matrixes  should  be  used  as  appropriate  to  introduce  the 
events  described.  The  structure  of  an  OT  differs  as  a  function 
of  its  complexity  and  can  be  made  up  of  a  training  phase,  pilot 
test,  several  scenario  driven  test  phases,  and  special  exercises 
or  excursions. 
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a.  Define  the  test  structure  and,  for  each  element,  specify 
what  is  to  be  included  and  how  it  is  to  be  accomplished.  Based 
on  the  test  approach  in  TEP  chapter  2,  describe  the  phases, 
trials,  and  test  events  to  be  executed. 

b.  A  test  trial  is  a  continuous  tactical  exercise  beginning 
from  systematically  controlled  initial  conditions  and  evolving 
tactically  within  some  controlled  bounds  until  some  specified 
duration  or  objective  has  been  met.  A  test  trial  is  the  smallest 
test  planning  unit  that  can  be  scheduled  to  occur  under  specific 
conditions  at  a  specific  time. 

c.  Test  trials  are  made  up  of  test  events,  which  are 
occurrences  about  which  data  are  required  (such  es  a  helicopter 
unmasking  from  a  predetermined  location  and  attempting  to  acquire 
targets  quickly  in  a  tactically  deployed  array) .  Other  times 
test  trials  consist  of  many  events  which  occur  under  different 
tactically  varied  conditions  within  the  trial  (for  example,  a 
free-play  force-on-force  exercise) . 

^  d.  Scenarios,  missions,  or  vignettes  are  built  up  from 

scheduled  trials  and  may  stretch  out  for  relatively  long  periods 
(for  example,  testing  of  an  infantry  weapon  through  multiple 
72-hour  field  exercises) . 

e.  Test  phases  refer  to  distinct  periods  of  time  within  which 
similar  scenarios  are  conducted,  possibly  on  ranges  separated  by 
substantial  distances  (such  as  force-on-force  phase  versus  live- 
fire  phase) .  Phases  usually  represent  the  largest  breakdown  of 
the  test. 

3-207.  Control  procedures 

Include  necessary  descriptions  of  the  control  structure  and 
procedures  that  will  be  used  to  ensure  that  required  test  events 
occur  in  situations  that  realistically  depict  the  tactical 
context  of  the  test  in  accordance  with  the  OMS/MP.  The  test 
design  ties  together  the  control  requirements  for  each  OTMOP. 

The  control  procedures  are  expanded  into  a  Control  Concept  in  the 
TEP  appendixes  and  into  a  Control  Plan  in  the  DTP.  Develop  an 
overall  schedule  of  events  as  the  basis  for  the  detailed  test 
schedule. 

3-208.  Overall  methodology 

Describe  execution  procedures,  control  procedures,  data 
collection  procedures,  and  any  other  information  necessary  for  an 
understanding  of  the  matrix  that  is  applicable  to  all  issues. 
Include  information  on  limitations  or  advantages  inherent  in  the 
test  concept  and  the  comparison  items  with  descriptions  of  the 
appropriate  characteristics  to  which  the  test  item  is  to  be 
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compared.  Include  provisions  to  ensure  that  test  events  adhere 
to  the  OMS/MP. 

3-209.  Test  limitations 

Describe  all  known  limitations  to  the  adequacy  of  the  test 
(e.g.,  duration,  number  of  systems,  availability  of  interoperable 
systems,  test  unit  availability,  and  ITTS)  and  their  impact  on 
the  OTMOP. 

3-210.  Test  details  ^  ^ 

Provides  details  of  issues,  events,  data  requirements,  and  data 
management.  Describe  methodologies  to  provide  the  OTMOP.  The 
organization  is  directly  aligned  with  the  issues  contained  in  TEP 
chapter  2 .  Each  issue  which  contains  one  or  more  OTMOP  is  an 
issue  to  be  addressed  in  OT.  For  abbreviated  evaluations  and  for 
testing  of  CBTDEV/TNGDEV  products,  all  of  the  MOP  developed  in 
chapter  2  will  be  OTMOP.  For  each  OTMOP,  develop  and  describe 
methodologies,  test  events,  control  concepts,  data  requirements, 
data  collection  and  reduction  procedures,  and  data  analysis 
techniques  peculiar  to  the  OTMOP.  If  the  methodologies  are 
closely  aligned  for  all  OTMOPs  associated  with  a  test  issue,  they 
need  only  be  stated  once.  If  not  closely  aligned,  separate 
paragraphs  must  be  developed  for  each  OTMOP. 

a.  OTMOP  specific  methodology.  Describe  the  test  events  that 
need  be  executed  to  generate  required  data,  the  operational 
conditions  under  which  the  events  must  occur,  the  techniques  for 
collecting  the  data,  and  the  control  procedures  that  will  be  used 
to  ensure  that  test  events  occur  at  the  proper  time  and  place  and 
under  the  conditions  specified.  Where  appropriate,  reference  can 
be  made  to  information  previously  presented  under  conduct  of 
test. 


(1)  Required  OTMOP  specific  test  events  should  be  listed  or 
referenced  to  the  appropriate  general  TEP  paragraph. 

(2)  Describe  the  OTMOP  specific  control  procedures  and 
rules  of  engagement  that  will  be  employed  to  ensure  that  required 
test  events  occur  in  situations  that  realistically  depict  the 
tactical  context  of  the  test.  Specify  the  level  of  operational 
realism  required  in  the  test  (full  tactical  simulation,  limited 
tactical  simulation,  or  no  tactical  simulation)  and  the  rationale 
for  the  choice. 

b.  OTMOP  specific  data  requirements.  Data  requirements 
applicable  to  each  OTMOP  should  be  extracted  from  the  POA  and 
summarized.  Present  information  concerning  data  accuracies  and 
sample  sizes  unless  otherwise  provided  in  the  general  TEP 
paragraphs . 
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c.  OTMOP  specific  data  collection.  Describe  the  procedures 
for  the  collection  and  tabulation  of  test  data  and  the  specific 
organization  and  procedures  to  be  used  to  collect  the  required 
data. 


d.  OTMOP  specific  data  reduction.  Describe  the  procedures 
for  the  processing  of  test  data.  Each  method  to  be  employed  to 
process  data  required  to  answer  the  OTMOP  should  be  addressed. 
Methods  include  manual,  instrumentation,  and  ADP.  Requirements 
such  as  equipment,  personnel  qualifications,  and  classification 
of  data  should  also  be  addressed. 

e.  OTMOP  specific  data  analysis.  Describe  the  logic  for 
formulating  and  computing  findings  and  any  assessments  which 
might  be  required.  The  POA  is  the  plan  as  to  how  the  data  to  be 
collected  during  test  will  be  processed,  analyzed,  or  combined  to 
answer  the  OTMOP.  This  is  an  extremely  important  step  in  the 
test  design  process  and  impacts  greatly  on  test  execution,  data 
reduction,  and  test  and  evaluation  report  writing. 

3-211.  Test  data  management 

The  test  designer  details  the  concepts  for  data  management  and 
the  requirements  for  the  tester  data  base.  The  data  management 
procedures  are  expanded  into  a  Data  Management  Concept  in  the  TEP 
appendixes  and  into  a  Data  Management  Plan  in  the  DTP. 

3-212.  Data  collection  concept 

Describes  the  test  element  organization  and  responsibilities  for 
data  collection  and  instrumentation  operation  and  maintenance. 
Delineate  the  requirements  for  data  collector  training  on  the 
test  item  and  use  of  data  collection  equipment.  Identify  the 
classification  level  of  the  test  data  (when  the  test  generates 
either  classified  data  (lAW  AR  380-5)  or  "Competition  Sensitive" 
data,  provisions  for  the  marking,  handling,  storage,  and 
disposition  are  to  be  addressed) .  Identify  procedure  by  which 
authorized  remote  users  can  access  the  data  along  wi  procedures 
for  documenting  and  archiving  the  data. 

3-213.  Data  reduction  concept 

Describe  the  process  in  which  recorded  test  data  is  organized, 
reduced,  verified,  managed,  controlled,  and  stored.  Organize  the 
data  in  terms  of  the  collection  source  (RAM  collection  form, 
cockpit  digital  recorder,  radar  tapes,  etc.).  Diagram  the 
process  through  which  each  set  of  collected  data  is  to  pass 
before  reaching  the  storage  medium  which  supports  the  test 
directorate  and  the  evaluator.  This  data  flow  diagram  identifies 
where  data  is  combined  with  other  data,  where  it  is  processed, 
scored,  reorganized,  validated,  or  otherwise  manipulated.  This 
includes  identifying  where  data  is  transferred  from  data  forms  to 
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automated  form.  Once  the  process  is  outlined,  describe  the 
actions  which  are  to  be  performed  on  the  data  at  each  step.  When 
judgement  of  the  test  directorate  or  DAG  is  required  at  any  step, 
the  rules  or  procedures  to  be  used  are  delineated.  After 
defining  this  process,  ADP  hardware,  software,  facilities  and 
personnel  requirements  are  refined,  and  the  OTP  is  updated. 

3-214.  Quality  control  concept 

Outline  process  for  independently  validating  test  data.  Define 
checks  and  procedures  to  be  used  to  preclude  or  detect  and 
correct  errors  made  in  data  collection,  data  entry,  or  data 
reduction.  Identify  emerging  data  summaries  required  to  identify 
potential  inconsistencies.  Outline  the  process  for  making 
required  corrections  or  changes  in  test  data  and  how  an  audit 
trail  for  those  corrections  or  changes  is  maintained.  Data 
verification  determines  whether  the  data  correctly  represent  the 
variables  they  characterize,  and  whether  the  data  collected  are 
adequate  to  support  the  test  and  evaluation  reports.  Specify 
requirements  for  and  techniques  proposed  for  data  verification. 
Data  collection  procedures  are  to  be  validated  prior  to  starting 
the  test  and  checked  during  the  test  to  insure  that  critical  data 
are  being  accurately  collected. 

3-215.  Data  base  design 

When  test  data  are  extensive  enough  to  require  storage  in  an 
automated  data  base,  describe  the  structure  and  content  of  the 
data  base.  Describe  the  architecture  and  design  of  the  data  base 
including  the  relationships  among  files  and  records.  Design  the 
architecture  to  support  the  view  of  the  data  required  by  the  TEP 
but  also  support  views  required  for  test  contrcl  and  data  quality 
control.  The  data  base  structure  and  data  dictionary  become  the 
core  of  the  data  base  description  in  the  test  and  evaluation 
report . 


a.  Data  element  dictionary.  The  data  in  each  file  or  record 
are  to  be  listed  and  augmented  by  any  necessary  definitions. 

Good  data  definitions  specify  exactly  what  is  measured  when. 
Examples  are: 

(1)  "Elapsed  time  to  transmit  a  call  is  to  be  measured  and 
recorded  to  the  nearest  second  by  a  data  collector  at  the 
transmitter  using  a  stopwatch.  Transmission  time  begins  when  the 
operator  . . .  <specify  operator  action>  and  ends  when  .... 

<specify  operation>." 

(2)  "RMS  display  range  is  the  RMS  range  between  the 
aircraft  and  the  ground  target  at  the  time  when  the  1553  data  bus 
in  the  aircraft  confirms  that  the  ground  target  symbol  is 
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displayed  on  the  aircraft  gunner's  scope.  This  is  available  from 
the  RMS  instrumentation  system." 

b.  Provide  data  base  structure  diagrams  which  expand  on  data 
base  descriptions  in  TEP  chapter  2 . 

c.  Provide  a  crosswalk  of  data  elements  and  the  data 
collection  forms  on  which  these  data  elements  will  be  collected. 

3-216.  Test  personnel,  equipment,  and  materials 

a.  Describe  the  makeup  and  general  organization  of  the  test 
directorate  required  to  execute  the  test  as  described  in  the 
test. 

b.  Describe  test  training  requirements.  Training 
requirements  (actual  training  and  necessary  resources)  consist  of 
two  major  elements;  training  the  test  players  and  training  the 
test  organization  personnel.  The  training  procedures  are 
expanded  into  a  training  concept  in  the  TEP  appendixes  and  into  a 
Training  Plan  in  the  DTP. 

(1)  A  summary  of  training  to  be  given  to  personnel  or  units 

pi^eparation  for  certification  (OTRS)  and  testing  should  be 

presented.  Some  means  of  training  are  an  ARTEP,  a  graded  FTX, 
special  schools,  and  new  equipment  training  teams  (NETT).  The 
specific  requirements  should  be  derived  from  the  Training  Suppor 
Package.  (See  Part  One.)  Contents  should  include  brief 
descriptions  of  the  skills  needed  and  the  expected  proficiency 
levels  of  both  typical  test  troops  and  the  OPFOR.  Also  indicate 
what  agency  (representative -of  the  Army  trainer)  will  furnish  the 
training  OTRSs. 

(2)  Training  to  be  presented  to  controllers,  data 
collectors  and  reducers,  and  ot  -r  test  personnel  should  also  be 
summarized.  Brief  statements  vcerning  special  schools,  data 
collection  and  reduction  methc  hands-on  practical  exercise, 
and  data  quality  control  are  ap;  opriate. 

c.  Describe  the  instrumentation  and  services  required  to 
support  the  test.  Instrumentation  consists  of  electronic  devices 
and  systems  designed  to  collect,  process,  and  document  test  event 
data.  Include  discussions  of  video  instrumentation,  existing 
instrumentation  systems  to  be  used,  design  and  construction  of 
new  instrumentation  devices  or  modification  of  existing 
instrumentation  devices  and  systems  required,  and  requirements 
for  SSED  for  computer  driven  systems.  Describe  the  use  of 
targets  and  simulators  both  existing  and  new.  The  ITTS 
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requirements  are  expanded  into  an  ITTS  Support  Concept  in  the  TEP 
appendixes  and  into  an  ITTS  Support  Plan  in  the  DTP. 

d.  Describe  required  audiovisual  support  to  the  test.  The 
audiovisual  requirements  are  expanded  into  an  Audiovisual  Support 
Concept  in  the  TEP  appendixes  and  into  an  Audiovisual  Support 
Plan  in  the  DTP. 

e.  Describe  the  architecture  of  the  ADP  support  to  the  test. 
The  ADP  requirements  are  expanded  into  an  ADP  Support  Concept  in 
the  TEP  appendixes  and  into  an  ITTS  Support  Plan  in  the  ADP. 

f.  List  other  materiel,  logistics,  personnel,  and 
communications  support  required  by  the  test.  The  test  support 
requirements  are  expanded  into  a  Test  Support  Concept  in  the  TEP 
appendixes  and  into  a  Test  Support  Plan  in  the  DTP. 

g.  List  base  and  range  facilities  (B&RF)  required  to  support 
the  planned  test. 

3-217.  Environmental  and  energy  impacts 
Summarize  the  environmental  and  energy  impacts  from  Test 
Environmental  Assessment  (TEA)  and  the  Environmental  Impact 
Statement  (EIS) . 


Section  XXVIII 
TEP  Appendixes 


3-218.  Purpose  of  appendixes 

The  appendixes  provide  needed  copies  of  supporting  documentation, 
provide  administrative  listings,  and  expand  TEP  paragraphs  into 
detailed  descriptions.  The  purpose  of  the  appendixes  is  to 
provide  complete  information  to  the  reader,  yet  avoid  interrupts 
in  the  logical  flow  of  information  in  the  main  body  of  the  TEP 
due  to  incorporation  of  too  much  detail.  For  this  reason,  many 
of  the  TEP  appendixes  are  optional  and  are  only  used  if  the 
information  in  the  particular  paragraph  of  the  TEP  is  of 
sufficient  volume  and  complexity  to  interrupt  logical  flow  of 
information. 

a.  Judicious  use  should  be  made  of  the  appendixes  to  avoid 
sidetracking  the  reader  of  the  TEP.  Reference  to  a  very  short 
appendix  is  wasteful.  The  information  should  be  included  in  the 
body  of  the  TEP.  If  too  much  information  is  included  in  the  body 
of  the  TEP,  the  reader  gets  bogged  down  with  detail.  In  this 
case,  the  information  should  be  summarized  in  the  body  of  the  TEP 
and  covered  in  detail  in  the  appendix.  In  no  case  should  the 
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appendix  duplicate  material  already  covered  in  the  body  of  the 
TEP. 


b.  For  full  evaluate  TEP,  the  evaluator  and  the  tester  share 
responsibilities  for  the  appendixes.  For  abbreviate  evaluate  TEP 
and  for  TEP  for  CBT/TNGDEV  products,  the  tester  prepares  all  the 
appendixes. 

3-219.  Content  of  the  appendixes 

a.  Supporting  documentation.  This  required  appendix  is 
developed  by  the  evaluator.  It  provides  a  listing  of  pertinent 
system  documentation  to  include,  as  a  minimum,  the  MNS,  ORD, 

TEMP,  OTP,  the  test  support  packages  and  the  TEA  and  EIS  (if 
applicable) .  Each  document  vill  be  listed  by  title,  responsible 
activity,  status  (e.g.  approved,  draft,  under  revision),  and 
dates  of  approval  or  date  of  draft.  See  Section  III  for 
literature  search  requirements.  Do  not  attach  copies  of  these 
documents  to  the  TEP  unless  called  out  in  a  succeeding  appendix. 

b.  Background.  This  optional  appendix  is  prepared  by  the 
evaluator  and  contains  details  of  the  background  of  the  system 
development  and  the  test  and  evaluation  of  the  system.  If  the 
details  to  be  placed  in  the  background  paragraph  of  the  TEP  are 
so  voluminous  that  continuity  of  the  TEP  is  lost,  they  should  be 
summarized  in  the  paragraph  and  stated  in  detail  in  appendix  B. 
Appendix  B  is  optional  if  the  details  can  be  adequately  covered 
in  the  background  paragraph.  (See  Section  VII.) 

c.  System  description.  This  optional  appendix  is  prepared  by 
the  evaluator  and  includes  a  description  of  the  system  (or 
combat/training  development  product) .  It  should  describe  the 
system  and  its  major  roles,  missions,  and  components  or 
characteristics.  A  description  of  similarities  and  differences 
between  the  system  under  test  and  the  objective  system  being 
developer*  s  appropriate  here.  The  concept  for  force  structure 
and  empl  nt  is  summarized.  If  a  large  amount  of  detail  is 
required  adequately  understand  the  system,  a  lengthy  system 
description  is  included  as  appendix  C.  Appendix  C  is  optional  if 
the  details  can  be  adequately  covered  in  the  system  description 
paragraph  of  the  TEP.  (See  Section  VII.) 

d.  Projected  threat.  This  optional  appendix  is  prepared  by 
the  evaluator  and  defines  the  approved  threat  in  the  post-IOC 
timeframe  of  the  tested  system  and  includes  capabilities,  typical 
means  of  operation,  and  known  methods  of  defeating  the  system. 

If  a  large  amount  of  detail  or  more  detail  is  required  to 
adequately  understand  the  threat,  a  lengthy  threat  statement  can 
be  included  as  appendix  D.  Appendix  D  is  optional  if  the  details 
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can  be  adequately  covered  in  the  threat  paragraph  of  the  TEP. 

See  Section  VII  and  Part  One  for  threat  details. 

e.  A  copy  of  the  DAG  Charter  and  SOP  is  included  by  the 
evaluator  if  requirements  for  a  DAG  are  specified  in  chapter  2  of 
the  TEP.  Details  for  the  preparation  of  a  DAG  charter  and  SOP 
are  contained  in  chapter  5.  For  abbreviated  evaluations  and  for 
tests  of  combat/ training  developments,  the  tester  may  charter  a 
DAG  and  add  this  appendix.  When  the  need  for  a  DAG  is  identified 
by  either  the  tester  or  the  evaluator,  its  mission  and 
responsibilities  are  defined  in  this  appendix. 

f.  A  copy  of  the  most  recent  and  approved  OTP  referenced  in 
appendix  A  is  included  as  appendix  F  by  the  tester.  Where 
appropriate,  a  test  resume  sheet  (TRS)  may  be  substituted  for  the 
OTP.  (See  chapter  4.) 

g.  The  POA  is  the  tester's  detailed  refinement  of  each 
operational  test  issue  and  associated  criteria  into  OTMOPs  and 
data  elements.  It  should  be  developed  as  the  first  step  in  the 
preparation  of  the  test  design  for  all  tests.  (See  Section  XVI 
for  POA  requirements  and  instructions.) 

h.  Test  scenario(s).  This  optional  appendix  contains  details 
of  the  test  scenario  not  contained  in  the  main  body  of  the 
document.  (See  Section  XXVII  for  details.) 

i.  Test  threat.  This  optional  appendix  contains  details  of 
threat  portrayal  not  contained  in  the  main  body  of  the  document. 
(See  Section  XXVII  for  details.) 

j .  Control  concept  appendix  includes  descriptions  of  the 
control  structure  and  procedures  that  will  be  used  to  ensure  that 
required  test  events  occur  in  situations  that  realistically 
depict  the  tactical  context  of  the  test  in  accordance  with  the 
QMS.  (See  Sections  XXIII  and  XXVII  and  chapter  5  for  details.) 
The  control  concept  is  a  preview  of  the  Control  Plan  in  the  DTP 
and  consists  of  three  parts: 

(1)  Level  of  operational  realism.  The  choice  among  the 
three  levels  of  operational  realism  should  be  stated  along  with  a 
short  rationale  for  the  choice.  This  may  be  dictated  by  the  TSP. 

(2)  Synopsis  of  events.  A  very  short  overview  of  the 
proposed  test  events  and  the  environment  and  sequence  in  which 
they  will  occur. 

(3)  Synopsis  of  control  methods.  An  overview  of  the 
methods  to  be  employed  in  causing  the  test  events  to  happen. 


t 
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k.  Data  management  concept  appendix  outlines 
organization  and  responsibilities  for  data  management.  The  data 
management  concept  is  a  preview  of  the  Data  Management  Plan  of 
the  DTP.  (See  Sections  XXIV  and  XXVII  and  chapter  5  for 
details.)  The  following  information  is  to  be  included. 


(1)  Collection  methods.  Each  method  of  data  collection 
intended  for  the  test  is  identified.  Under  each  method  is  listed 
the  general  categories  of  data  it  will  be  used  to  collect. 


(2)  Collection  support.  Any  special  requirements  _ 
pertaining  to  data  collection  are  identified.  Examples  include 
an  overview  of  test  organization  and  planned  initial  quality 
control  procedures,  instrumentation  requirements,  data  collectors 
with  special  qualifications,  expected  security  and  special  data 
handling  requirements,  or  video  and  photography  documentation. 

(3)  Reduction  methods.  The  concept  through  which  recorded 
test  data  is  organized,  reduced,  verified,  managed,  controlled, 
and  stored.  Each  data  requirement  and  its  means  of  collection 
are  delineated  in  the  body  of  the  TEP. 


(4)  Data  presentation.  Identify  the  concept  for  data 
display  requirements  including  both  management  displays  and  test 
report  displays. 

(5)  Data  base  design.  Describe  the  concept  for  structure 
and  content  of  the  automated  data  base.  If  available  at  this 
time,  the  description  of  the  test  data  base  will  include: 


( 

labels,  a 
each  file, 
automated . 


a)  Identification  of  the  required  files  to  include 
short  description  of  the  type  of  data  to  be  stored  in 
and  an  indication  for  each  file  whether  it  is  to  be 


(b)  A  description  of  the  architecture  used  to  organize 
the  data  base. 


(c)  Definition  of  data  elements  in  each  file  to  include 
sufficient  definitions  to  preclude  misunderstanding  and  ambiguity 
(Data  Element  Dictionary) . 


(6)  Data  base  management  paragraph  delineates  the 
procedures  for  accessing  and  changing  the  data  base.  The 
activities  or  individuals  who  can  query,  manipulate,  view,  or 
■odify  all  or  part  of  the  data  base  or  other  test  results  will  be 
delineated.  The  authority  to  release  all  or  part  of  the  data 
will  also  be  specified. 
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(7)  Quality  control  concept  paragraph  outlines  the  concept 
for  independently  validating  test  data.  It  defines  the  checks 
and  procedures  which  are  to  be  used  by  the  tester  to  preclude  or 
detect  and  correct  errors  made  in  data  collection,  data  entry,  or 
data  reduction. 

l.  ITTS  support  concept.  This  optional  appendix  contains 
details  of  planned  ITTS  not  contained  in  the  main  body  of  the 
document.  It  is  expanded  into  an  ITTS  Support  Plan  in  the  DTP. 
(See  chapter  5.) 

m.  Audiovisual  support  concept.  This  optional  appendix 
contains  details  of  planned  audiovisual  support  not  contained  in 
the  main  body  of  the  document.  It  is  expanded  into  an 
Audiovisual  Support  Plan  in  the  DTP.  For  full  evaluation  systems 
audiovisual  coverage  is  required.  (See  chapter  5.) 

n.  ADP  support  concept.  This  optional  appendix  contains 
details  of  planned  ADP  support  not  contained  in  the  main  body  of 
the  document.  It  is  expanded  into  an  ADP  Support  Plan  in  the 
DTP.  (See  chapter  5.) 

o.  The  training  concept  is  actually  developed  during  the 
evolution  of  the  control  and  data  management  plans.  It  is 
expanded  into  a  Training  Plan  in" the  DTP.  (See  Sections  XXV  and 
XXVII  and  chapter  5  for  details.)  The  following  guidance  should 
be  considered  for  each  group  to  be  trained: 

(1)  Training  of  player  personnel  will  provide  individual, 
section,  and  unit  level  skills  necessary  to  operate  and  maintain 
the  equipment  or  perform  the  tactics  to  be  tested. 

(2)  In  those  tests  where  OPFOR  personnel  is  included,  the 
units  portraying  the  enemy  must  be  trained  to  employ  the  tactics 
of  the  threat  force.  The  employment  of  U.S.  tactics  and 
procedures  by  OPFOR  may  invalidate  or  cast  doubts  on  results  of 
such  a  test. 

(3)  As  a  minimum,  controllers  must  be  familiar  with  the 
control  plan  and  the  data  management  plan.  They  must  be  capable 
of  making  sound  independent  decisions  on  control  matters  in  the 
absence  of  communications  and  should  be  able  to  act  as  data 
collectors  should  the  need  arise.  They  should  also  be  able  to 
implement  emergency  procedures. 

(4)  Training  for  data  collectors  will  include  instruction 
on  the  control  plan  and  the  data  management  plan.  Particular 
emphasis  should  be  given  to  the  need  for  accuracy  and  unbiased 
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“coirroU«r  an^l^pleienf  ?f  nUessary. 

personili 

and  quality  control  procedures.  resources  must  be  planned 

“SSI  -“.ssa  ::r„:s.“.s  ■>» 

provide  the  best  preparation. 

(6)  Test  directorate  and  DAG  personnel  lMt?uctioiI 

irisiSl^'necfsiarrand  ?rainin,  time  and  other  resources  »ust 
be  planned  accordingly. 

o  Test  support  concept.  This  ?P^^°”^gj^^^faciiities^not 
details  i®®^a7?f’'thrdMlIsent.  It  is  expanded  into 

neiriuppSrl  wan  ?n  the  DTP.  (See  chapter  5.) 

(1)  Test  support  “‘^'^resses^the^categori  for  the 

administrative,  J  activities.  ^It  includes  general 

irp^ortTuncfion^^^^^^ 

administrative  requirements. 

(2)  AS  for  the  other  support  planning^requirenents.^the^^^ 

fn^jrcuSln^n^rrfnrpi-inrif 

collection/reduction  requirements. 

<3,  The  test  support  plan  f„--"”t-h"res“ 
items  including  but  not  g  Seating  and  air  conditioning, 

including  rental  cars,  gen  ^  '  procured  equipment  such  as 

special  fabrication  items,  ,  military  standard 

copying  machines  and  suoDort^Cinr  Ung  offices,  shelters, 

items);  tv^  telephonL  and  ot  communications 

^^I^re:4.n?fnS^“‘nS'anynlcessary« 

Sf  devSiop^^ir'deiitery  of  test  support  itess  and  the  use  o 
lacim!es,  ranges,  and  equipment. 
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equipment  are  fully  trained.  Tests  posing  unusual  or  potentially 
hazardous  conditions  that  involve  risk  beyond  the  normal  call  of 
duty  require  approval  by  the  Surgeon  General  and  are  to  be 
conducted  within  the  provisions  of  AR  70-25.  The  system  safety 
release  is  also  included  or  referenced. 

(5)  The  electronic  warfare  plan  is  a  detailed  set  of 
instructions,  procedures,  and  schedules  for  those  portions  of  the 
test  involving  electronic  warfare.  It  is  applicable  when 
electronic  warfare  is  a  major  condition  of  the  test  but  not  the 
purpose  or  major  objective. 

(6)  The  visitor  control  instructions  document  requirements 
for  visitor  control,  briefings,  security  clearance  verifications, 
conferences,  field  tours,  transportation,  special  equipment  (such 
as  laser  goggles,  boots,  and  field  pants) ,  and  other  related 
details. 

(7)  The  public  affairs  plan  describes  the  measures  to 
process  requests  or  actions  from  the  news  media  and  local 
populace.  Paragraph  3-29,  AR  360-5,  sets  the  Army  policy  on 
media  coverage  of  Army  test  activities.  It  states:  "The 
Department  of  the  Army  does  not  permit  media  coverage  of 
developmental,  technology  validation,  or  operational  testing  of 
Army  systems."  Consult  with  the  local  public  affairs  officer  for 
details. 

(8)  The  OPSEC  plan  addresses  the  sensitive  aspects  for  the 
test,  the  hostile  threat,  the  vulnerabilities,  countermeasures, 
OPSEC  training,  and  priorities  of  the  test.  If  appropriate  for 
the  test's  classification,  the  test  officer  must  plan  for  the 
protection  of  such  information  as  communications  (document  or 
electronic  means),  informative  patterns  and  signatures  (visual, 
acoustic,  electronic,  or  infrared),  and  stereotyped  procedures 
(tactical,  physical,  or  administrative). 

(9)  The  security  plan  encompasses  all  activities  required 
to  ensure  internal  physical  security  at  the  headquarters  area  and 
the  security  of  the  test  site  along  with  their  associated 
property  to  include  the  tested  system.  Coordinate  this  plan  with 
your  security  manager.  The  security  manager  should  also  be 
consulted  on  access  badges  and  security  clearances. 

q.  A  copy  of  the  approved  Test  Environmental  Assessment  (TEA) 
is  included  by  the  tester. 

r.  A  copy  of  the  approved  Environmental  Impact  Statement 
(EIS)  is  included  by  the  tester.  Planning  for  reduction  of 
impact  on  the  environment  must  be  conducted  for  all  tests  in 
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which,  as  a  result  of  test  requirements,  a  potential  for  harm  to 
the  environment  exists.  This  planning  is  test  dependent  but 
generally  concerns  areas  of  land,  water,  and  air  use  or 
endangerment  of  animals,  birds,  or  water  life.  In  many  cases,  a 
formal  environmental  impact  statement  or  environmental  assessment 
must  be  obtained  before  the  test  can  be  conducted.  The  test 
officer  should  identify  as  early  as  possible  whether  a 
requirement  for  an  environmental  assessment  exists  and  coordinate 
an  initial  determination  of  actions.  The  environmental 
assessment  and  procedures  to  be  employed  during  the  test  to 
reduce  environmental  impacts,  if  required,  are  documented  in  the 
EIS  with  appropriate  information  brought  forward  to  chapter  3  of 
the  TEP. 

s.  DOTE  approval  of  the  test  concept  is  required  for  TEP  for 
all  DOTE  oversight  programs.  The  evaluator  will  insert  a  copy  of 
the  formal  DOTE  approval  of  the  test  concept.  (See  Paragraph  3- 
13  and  Part  One.) 

t.  Test  change  proposals  (TCP) .  This  required  appendix  is 
added  as  a  placeholder  by  the  tester.  As  TCP  are  developed 
during  the  time  between  approval  of  the  TEP  and  the  end  of  the 
test,  they  are  added  in  chronological  sequence  as  tabs  the 
appendix  and  are  listed  in  the  body  of  the  appendix. 

(1)  TCPs  are  required  for  any  changes  to  an  approved  TEP. 

A  TCP  will  be  initiated  any  time  a  change  to  the  approved  TEP  is 
proposed  by  the  tester,  evaluator,  or  other  official.  It  will 
outline  the  details  of  the  proposed  change  and  estimate  the 
impact  on  test  resources. 

(2)  TCPs  for  TEPs  for  full  evaluate  level  systems  will  be 

submitted  in  the  format  shown  in  figure  3-43.  TCPs  for  TEPs  for 
other  than  full  evaluate  level  systems  will  be  submitted  in  the 
format  shown  in  figure  3-44.  Attachments  to  the  TCP  should 
consist  of  changed  res  to  be  inserted  in  the  TEP,  changed  cost 

estimates,  or  other  terial  as  appropriate  to  document  the 

recommended  change. 

(INSERT  FIGURE  3-43) 

(INSERT  FIGURE  3-44) 

(3)  The  change  should  not  be  implemented  until  approved, 
but  approval  of  immediate  TCP  requirements  need  not  wait  for 
formal  documentation.  The  test  directorate  should  notify  the 
appropriate  personnel  that  a  TCP  is  required  and  the  reason  for 
the  requirement.  This  should  be  performed  telephonically  or 
electronically  (e.g.,  by  e-mail).  Necessary  action  to  obtain 


3-102 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


verbal  or  conditional  approval  can  be  performed  based  upon  this 
early  notification.  Formal  documentation  can  be  prepared  as 
required. 

(4)  Approval  authority  for  the  TCP  is  the  same  level  as  for 
the  TEP. 

(5)  If  possible,  concurrence  of  the  tester  and  evaluator, 
if  appropriate,  should  be  obtained  on  the  TCP  itself.  A  method 
of  accomplishing  this  is  shown  in  figure  3-45.  This  block  should 
be  inserted  in  the  coordination  paragraph  of  the  TCP  just  prior 
to  the  approval  block. 

(INSERT  FIGURE  3-45) 

u.  Glossary,  Acronyms,  and  Abbreviations  appendix  is  prepared 
as  required  by  the  evaluator  and  the  tester.  Any  unusual^ 
technical  terms  or  frequently  used  acronyms  and  abbreviations  in 
the  body  of  the  TEP  or  in  other  appendixes  will  be  defined  in 
this  appendix  by  the  evaluator  for  his  portion  of  the  TEP  and  by 
the  tester  for  those  terms  which  he  has  introduced  and  which  were 
not  previously  defined  by  the  evaluator. 

V.  TEP  coordination  record  is  a  listing  of  agencies  with 
which  the  TEP  was  coordinated  and  a  record  of  the  resolution  of 
their  comments.  (See  Paragraphs  3-12  and  3-16.) 

w.  For  historical  purposes,  all  authors  and  supporting 
personnel  having  input  to  or  knowledge  of  the  writing  of  the  TEP 
should  be  listed  in  this  appendix  by  the  evaluator. 

X.  Distribution  of  the  TEP  by  organization  and  number  of 
copies  is  shown  in  this  appendix  by  the  evaluator.  (See 
Paragraphs  3-12  and  3-16.) 

3-220.  Documentation  of  appendixes 

Appendixes  are  included  in  the  TEP  as  shown  in  figure  3-46. 

(INSERT  FIGURE  3-46) 
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Figure  3-1 .  Operational  test  and  evaluation  cycle. 


TEST  AND  EVALUATION  PLAN  (TEP)  FORMAT 

CHAPTER  1.  INTRODUCTION 

1.1.  Purpose  of  the  Evaluation 

1.2.  Scope  of  the  Evaluation 

1.3.  Background 

1.3.1.  Program  Background 

1.3.2.  Test  and  Evaluation  Background 

1.3.3.  COEA/OTE  Relationship 

1.4.  System  Description 

1.5.  Projected  Threat 

1.6.  Test  and  Evaluation  Milestones 

1.7.  Operational  Issues  and  Criteria  (OIC) 

1.7.1.  COIC 

1.7. 1.1.  COIC  1  Issue  Statement 

1.7. 1.1.1.  COIC  1  Scope 

1.7. 1.1. 2.  COIC  1  Criteria 

1.7. 1.1. 3.  COIC  1  Rationale 

1.7. 1.2.  COIC  2 
through 

1.7. 1. m.  COIC  m 

1.7.2.  AOIC 

1.7.2. m+l.  AOIC  m+1  Issue  Statement 

1.7.2. m+l.l.  AOIC  m+1  Scope 

1.7. 2.  m+1. 2.  AOIC  m+1  Criteria 

1.7. 2.  m+1. 3.  AOIC  m+1  Rationale 

1.7.2. m+2.  AOIC  m+2 
through 

1.7.2. n-m.  AOIC  n 

1.7.3.  Baseline  Correlation  Matrix  (BCM) 

CHAPTER  2.  EVALUATION  PLAN  (ANALYSIS  PLAN) 

2.1.  Evaluation/Analytic  Approach 

2.1.1.  Overview  of  Evaluation  (Analytic)  Approach 

2.1.2.  Aggregation  of  Operational  Effectiveness 

2.1.3.  Aggregation  of  Operational  Suitability 

2.2.  Operational  Issues  (from  1.7) 

2.2.1.  Issue  1 

2.2. 1.1.  Criterion  1-1 

2. 2. 1.1.1.  MOE/MOP  1-1-1 

a.  Analytic  Design 

b.  Analytic  Procedures 

c.  Presentations 

2. 2. 1.1. 2.  MOE/MOP  1-1-2 
through 

2.2.1.1. x.  MOE/MOP  1-1-x 

2.2.1.1. X+1.  Criterion  Analytic  Summary 


Figure  3-2.  TEP  format 


TEST  AND  EVALUATION  PLAN  (TEP)  FORMAT  (continued) 

2. 2. 1.2.  Criterion  1-2 
through 

2.2. 1. y.  Criterion  1-y 

2.2.1. y+l.  Issue  Analytic  Summary 

2.2.2.  Issue  2 
through 

2.2. n.  Issue  n 

2.3.  Data  Source  Matrix  (DSM) 

2.4.  User  Test  Approach 

2.4.1.  Test  Scope 

2.4.2.  Factors  and  Conditions 

2.4.3.  Test  Design  Matrix (es) 

2.4.4.  Sample  Sizes 

2.4.5.  Requirements  for  a  DAG 

2.5.  Other  Sources  of  Data 

2.5.1.  Other  User  Test  (Repeat  as  necessary) 

2 . 5 . 1 . 1 .  Test  Scope 

2 . 5 . 1 . 2 .  Factors  and  Conditions 

2.5.2.  DT  or  Contractor  Test  (Repeat  as  necessary) 

2 . 5 . 2 . 1 .  Test  Scope 

2. 5. 2. 2.  Factors  and  Conditions 

2.5.3.  Modeling/Simulation  (Repeat  as  necessary) 

2. 5. 3.1.  Model  (Simulation)  Description 

2. 5. 3. 2.  Verification  and  Validation 

2. 5. 3. 3.  Accreditation 

2.5.4.  Market  Survey/Investigation 

2.5.5.  Other  Studies  or  Analyses 

2.6.  Data  Base  Structure 

2.6.1.  Identification  of  Required  Files 

2.6.2.  Description  of  File  Relationships 

2.6.3.  Data  Element  Definitions 

2.7.  Evaluation  Limitations 


Figure  3-2.  TEP  format  (cont) 
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3 

3 
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.2 
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CHAPTER  3.  TEST  DESIGN 

3.1.  Test  Description 

1.1.  Test  Purpose 

1.2.  Test  Authority 

1.3.  Test  Overview 

3. 1.3.1.  Test  Phases 

3. 1.3. 2.  Factors  and  Conditions 
Conduct  of  Test 

2.1.  Tactical  Context 

3. 2. 1.1.  Scenario(s) 

3 . 2 . 1 . 2 .  Test  Environment 

3.2. 1.3.  Threat  for  Test 

3.2. 1.4.  Tactics  and  Doctrine 

3. 2. 1.5.  Test  Unit  Organization 

3.2.2.  Test  Events 

3. 2. 2.1.  Mission  and  Vignette/Battle  Description 

3. 2. 2. 2.  Site  Specific  Vignettes/Battles 

3. 2. 2. 3.  Vignette/Battle  List 

3. 2. 2. 4.  Event  Matrix (ces) 

3 . 2 . 2 . 5 .  Event  Sequencing 

3.2.3.  Control  Procedures 

3.2.4.  Schedule  of  Events 

3.2.5.  Overall  Methodology 

3.2.6.  Test  Limitations 
.3.  Test  Details 

3.3.1.  Issue  1 

3.3. 1.1.  OTMOP  1-1 

a.  Methodology  (OTMOP  Peculiar) 

(1)  Test  Events 

(2)  Control  Concepts 

b.  Data  Requirements  (OTMOP  Peculiar) 

c.  Data  Collection  (OTMOP  Peculiar) 

d.  Data  Reduction  (OTMOP  Peculiar) 

e.  Data  Analysis  (OTMOP  Peculiar) 

3.3. 1.2.  OTMOP  1-2 
through 

3.3.1.x.  OTMOP  1-x 

3.3.2.  Issue  2 
through 

3.3.n  Issue  n 


Figure  3-2.  TEP  format  (cont) 
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TEST  AND  EVALUATION  PLAN  (TEP)  FORMAT  (continued) 

3.4.  Test  Data  Management 

3.4.1.  Data  Collection  Concept 

3.4.2.  Data  Reduction  Concept 

3.4.3.  Quality  Control  Concept 

3.4.4.  Data  Reduction  Timelines 

3.4.5.  Data  Base  Design 

3. 4. 5.1.  Data  Element  Dictionary 

3 . 4 . 5 . 2 .  Data  Base  Structure 

3. 4. 5. 3.  Data  Collection  Forms  Matrix 

3.5.  Test  Personnel,  Equipment,  and  Materials 

3.5.1.  Test  Directorate  Organization 

3.5.2.  Training 

3. 5. 2.1.  Test  Player  Personnel 

3 . 5 . 2 . 2 .  Test  Directorate  Personnel 

3.5.3.  Equipment  and  Materials 

3. 5. 3.1.  Instrumentation 

3 . 5 . 3 . 2 .  Targets  and  Simulators 

3 . 5 . 3 . 3 .  ADP  Equipment 

3 . 3 . 3 . 4 .  Audiovisual  Support 

3 . 3 . 3 . 5 .  Test  Support  Equipment 

3. 3. 3. 6.  Test  Support  Facilities 

3.6.  Environmental  and  Energy  Impacts 


Figure  3-2.  TEP  format  (cont) 
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APPENDIX 

S. 

DOT&E  Test  Concept  Approval 

APPENDIX 

T 

Test  Change  Proposals 

TAB  1. 

Test  Change  Proposal  #1 

1  through 

1  TAB  n.  Test  Change  Proposal  #n 

APPENDIX 

u. 

Glossary,  Acronyms,  and  Abbreviations 

APPENDIX 

V. 

TEP  Coordination  Record 

APPENDIX 

w. 

Authors  and  Supporting  Personnel 

APPENDIX 

X. 

TEP  Distribution  List 

Figure  3-2.  TEP  format  (cont) 
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TEP 

Section 

TEST  AND  EVALUATION  PLAN  (TEP) 
PREPARATION  RESPONSIBILITIES  MATRIX  ' 

TYPE  OF  TEST /MANAGEMENT  LEVEL  OF  SYSTEM 

Full  Abrv  Prop  CEP 
Eval  Eval  FDTE  Test 

Cust 

Test 

Eval 

Onlv 

CHAPTER  1 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.1 

Purpose 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.2 

Scope 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.3 

Background 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.4 

Sys  Descript 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.5 

Proj  Threat 

EUI 

TUI 

TUI 

TUI 

TUI 

EUI 

1.6 

T&E  Milestones 

ETI 

TEI 

TUI 

TUI 

TUI 

E 

1.7. 

1  COIC 

EUI 

TUI 

TUI 

TUI 

TUI 

EUI 

1.7. 

2  AOIC 

E 

TEI 

TUI 

TUI 

TUI 

E 

1.7. 

3  BCM 

E 

NR 

NR 

NR 

NR 

E 

CHAPTER  2 

E 

T 

T 

T 

T 

E 

2.1 

Eval /Anal  Approach 

E 

T 

T 

T 

T 

E 

2.2 

Operational  Issues 

E 

T 

T 

T 

T 

E 

2.3 

Data  Source  Matrix 

E 

NR 

NR 

NR 

NR 

E 

2.4 

User  Test  Aproach 

E 

NR 

NR 

NR 

NR 

NR 

2.5 

Other  Data  Sources 

E 

NR 

NR 

NR 

NR 

E 

2.6 

Data  Base  Struc 

E 

NR 

NR 

NR 

NR 

E 

2.7 

Eval  Limitations 

E 

NR 

NR 

NR 

NR 

E 

CHAPTER  3 

T 

T 

T 

T 

T 

E 

3.1 

Test  Description 

T 

T 

T 

T 

T 

NR 

3.2 

Conduct  of  Test 

T 

T 

T 

T 

T 

NR 

3.3 

Test  Details 

T 

T 

T 

T 

T 

NR 

3.4 

Test  Data  Mgmt 

T 

T 

T 

T 

T 

NR 

3.5 

Pers-Equip-Matl 

T 

T 

T 

T 

T 

NR 

3.6 

Envir/ Energy  Impact 

1 

T 

T 

T 

T 

NR 

1  Legend 

E  = 

aluator,  T  =  Tester,  U 

=  User, 

I  = 

Input , 

NR  = 

Not 

Exai 

-  equired 

le:  TEI  =  Tester  responsibility 

,  Evaluator  Input. 

Figure  3-3.  TEP  preparation  matrix 
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APPENDIXES  T 

A  Supporting  Doc  E 

B  Background  (opt)  E 

C  Sys  Descript  (opt)  E 

D  Proj  Threat  (opt)  EUI 

E  DAG  charter  and  SOP  E 

F  OTP  T 

G  Pattern  of  Analysis  T 

H  Test  Scenarios  (opt)  T 

I  Test  Threat  (opt)  T 

J  Control  Concept  T 

K  Data  Mgmt  Concept  T 

L  ITTS  Spt  Concept  T 

M  A-V  Spt  Concept  T 

N  ADP  Spt  Concept  T 

O  Training  Concept  T 

P  Test  Spt  Concept  T 

Q  Test  Environ  Assess  T 

R  Environ  Impact  State  T 


T 

T 

T 

T 

E 

T 

T 

T 

T 

E 

NR 

NR 

NR 

NR 

E 

T 

T 

T 

T 

E 

TUI 

TUI 

TUI 

TUI 

EUI 

NR 

TUI 

NR 

NR 

NR 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 


T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 


T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 


T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 

T 


S 

DOTE  TC  Approval 

E 

NR 

NR 

NR 

NR 

T 

Test  Change  Prop 

TEI 

T 

TUI 

TUI 

TUI 

U 

Glossary 

E 

T 

T 

T 

T 

V 

TEP  Coord  Record 

ETI 

T 

T 

T 

T 

W 

Authors 

ETI 

T 

T 

T 

T 

X 

TEP  Distrib  List 

ETI 

T 

TUI 

TUI 

TUI 

E 

E 

E 

E 


Legend: 

E  =  Evaluator,  T  =  Tester,  U  =  User,  I  =  Input,  NR  =  Not 
required 

Example:  TEI  =  Tester  responsibility.  Evaluator  Input. 


Figure  3-3.  TEP  preparation  matrix  (cont) 


MILESTONES  FOR  FULL-EVALUATE  LEVEL  TEP  DEVELOPMENT 


Action _ Resp _ Desired  Latest 


AUU-LUri 

Draft  COIC  Submitted 

CBTDEV 

J.  J. 

T-990 

xjci 

T-630 

COIC  Approved 

TRADOC 
or  DA 

T-930 

T-570 

TEMP  Approved 

MATDEV 
or  DOD 

T-870 

T-510 

Draft  TSP  to  Tester 

CBTDEV 

TNGDEV 

MATDEV 

ASAP 

T-480 

OTP  Submitted  to  TEXCOM  and 

Working  Group  TSARC 

Tester 

T-840 

T-480 

Draft  TEP  Ch  1  and  2  to  Tester 
(Eval  Dir  Signature) 

Eval 

ASAP 

T-435 

OTP  Submitted  to  OPTEC 

Working  Group  TSARC 

OPTEC 

T-750 

T-390 

OTP  Submitted  for  GO 

TSARC  Approval 

OPTEC 

T-720 

T-360 

Draft  TEP  Para  3.1  (Test  Concept) 
to  TEXCOM  and  OEC  for  Cmt 

Tester 

ASAP 

T-285 

Final  TSP  to  Tester 

CBTDEV 

TNGDEV 

MATDEV 

ASAP 

T-270 

OTRR  #1 

Tester 

T-270 

T-270 

Test  Concept  Pre-Brief 
to  DUSA(OR) 

OPTEC 

T-26 

Test  Concept  Briefing 
to  DOTE 

OPTEC 

T-250 

OEC  Approved  TEP  Ch  1  and  2 
(OEC  Staffing  Complete) 

Provided  to  Tester 

Eval 

ASAP 

T-230 

Figure  3-4.  Milestones  for  Full  Evaluate  TEP  Development 
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Action 

Reso 

Desired 

Latest 

Draft  TEP  Ch  3  to  TEXCOM 
and  DEC  for  Cmt 

Tester 

ASAP 

T-230 

Draft  TEP  Coord  with 

TIWG  Members 

Tester 

ASAP 

T-200 

TIWG  Members  Return 

Coord  Draft  Cmts 

TIWG 

Members 

T-180  1 

Final  Draft  TEP  to 

OEC/TEXCOM  for  Approval 

Eval  & 
Tester 

ASAP 

T-165  1 

TEP  Approved  by  DEC 
and  TEXCOM 

OEC  & 
TEXCOM 

ASAP 

T-150 

TEP  Approved  by  OPTEC 

OPTEC 

ASAP 

T-135 

Brief  Approved  TEP 
to  DUSA(OR) 

OPTEC 

T-130 

T-130 

Brief  Approved  TEP 
to  DOTE 

OPTEC 

T-120 

T-120 

TEP  to  DOTE  for  Approval 

1  of  Test  Concept  and  Design 

OPTEC 

T-120 

T-105 

1  Draft  DTP  Complete 

Tester 

ASAP 

T-90 

1  Strawman  TER  Complete 

Eval  & 
Tester 

T-90 

OTRR  #2 

OPTEC 

T-60 

1 

0^ 

o 

Final  DTP  Approval 

TEXCOM 

ASAP 

T-45 

1  Complete  Pilot  Test 

Tester 

T-3 

1  OTRR  #3 

OPTEC 

T-2 

T-1 

1  Test  Start 

Tester 

T-Date 

Figure  3-4  (cent) .  Milestones  for  Full  Evaluate  TEP  Development 
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DISTRIBUTION  LIST  FOR  TEPs 

(Materiel  Systems)  Number  of 

copies  per  type  TEP 
Draft  Approved 

Aaencv  TEP  TEP  . 

Deputy  Under  Secretary  of  the  Army 

1  (Operations  Research) 

1 

—  — 

1  Operational  Evaluation  Command 

2 

2 

1  Proponent 

1 

1 

TSM 

1 

1 

Coproponent 

1 

1 

Project  Manager 

1 

1 

Materiel  Developer 

1 

1 

USA  Air  Defense  Artillery  School 

1* 

1* 

USA  Armor  School 

1* 

1* 

USA  Aviation  School 

1* 

1* 

USA  Chaplain  Center  and  School 

1* 

1* 

USA  Chemical  School 

1* 

1* 

USA  Field  Artillery  School 

1* 

1* 

USA  Engineer  School 

!• 

1* 

USA  Infantry  School 

1* 

1* 

USA  Army  Institute  of  Pers  and  Resource 

Mgt  1* 

1* 

USA  Intelligence  Center  and  School 

1* 

1* 

USA  Intelligence  School 

1* 

1* 

USA  Military  Police  School 

1* 

1* 

USA  Missile  &  Munitions  Center  &  School 

1* 

1* 

USA  Ordnance  Center  and  School 

1* 

1* 

USA  Quartermaster  School 

1* 

!• 

USA  Signal  School 

1* 

1* 

USA  Transportation  School 

1* 

1* 

USASSC  and  Fort  Ben  Harrison 

1* 

1* 

USA  Combined  Arms  Center  and  Ft  Leavenworth  1* 

1* 

USA  Logistics  Center 

1* 

1* 

TEXCOM 

Methodology  and  lysis  Directorate 

2 

1 

Operations  Dire  rate 

1 

— 

Army  Research  Institute 

1 

1 

AMC  Liaison  Office 

1 

1 

Information  Systems  Directorate 

1* 

1* 

Operations  Support  Directorate 

1* 

1* 

Personnel,  Admin,  and  Logistics  Dir 

1* 

1* 

Resource  Management  Directorate 

1* 

1* 

•OPTEC  evaluation/ test  directorates  will  determine  if 
is  appropriate  for  each  specific  evaluation/test . 

staffing 

Figure  3-5.  Distribution  list  for  TEPs 
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DISTRIBUTION  LIST  FOR  TEPs  (cont) 

(IKA  Systems)  Number  of 

copies  per  type  TEP 
Draft  Approved 

TEP _ TEP 


Deputy  Under  Secretary  of  the  Army 

(Operations  Research)  1 

Operational  Evaluation  Command  2  2 

Proponent  ^  ^ 

Co-proponent  (If  applicable)  1  ^ 

Project  Manager  ^  ^ 

Materiel  Developer 

USA  Air  Defense  Artillery  School  l 

USA  Armor  School 

USA  Aviation  School 

USA  Chaplain  Center  and  School 

USA  Chemical  School 

USA  Field  Artillery  School  1| 

USA  Engineer  School 

USA  Infantry  School  i* 

USA  Army  Institute  of  Pers  and  Resource  Mgt  1* 

USA  Intelligence  Center  and  School  1*  1^ 

USA  Intelligence  School  1] 

USA  Military  Police  School 

USA  Missile  &  Munitions  Center  &  School  1|  1^ 

USA  Ordnance  Center  and  School  1^ 

USA  Quartermaster  School  1| 

USA  Signal  School 

USA  Transportation  School  1*  ^ 

USA  Information  Systems  Command  la  la 

USA  Information  Systems  Engineering  Command  la  la 

USA  Information  Systems  Support  Center  la  la 

USA  Logistics  Evaluation  Agency  la  la 

USASSC  and  Fort  Ben  Harrison  1*  1^ 

USA  Combined  Arms  Center  and  Ft  Leavenworth  1*  1^ 

USA  Combined  Arms  Support  Command  1*  1* 

TEXCOM 

Methodology  and  Analysis  Directorate  2  1 

Operations  Directorate  1 

Army  Research  Institute  1  1 

AMC  Liaison  Office  1  ^ 

Information  Systems  Directorate  1  l 

Operations  Support  Directorate  ^  1*  1^ 

Personnel,  Admin,  and  Logistics  Dir  1*  1^ 

Resource  Management  Directorate  1*  1 

•OPTEC  evaluation/ test  directorates  will  determine  if  staffing 
is  appropriate  for  each  specific  evaluation/test. 


MILESTONES  FOR  ABBREV- EVALUATE 

LEVEL  TEP 

DEVELOPMENT 

Action 

Re  so 

Desired 

Latest 

Draft  COIC  Submitted 

CBTDEV 

T-990 

T-630 

COIC  Approved 

TRADOC 
or  DA 

T-930 

T-570 

TEMP  Approved 

MATDEV 
or  DOD 

T-870 

T-510 

Draft  TSP  to  Tester 

CBTDEV 

TNGDEV 

MATDEV 

ASAP 

T-480 

OTP  Submitted  to  TEXCOM  and 

Working  Group  TSARC 

Tester 

T-840 

T-480 

AOIC  and  Evaluator  Concerns 

Provided  to  Tester 

Eval 

ASAP 

T-435 

OTP  Submitted  to  OPTEC 

Working  Group  TSARC 

OPTEC 

T-750 

T-390 

OTP  Submitted  for  GO 

TSARC  Approval 

OPTEC 

T-720 

T-360 

Draft  TEP  Para  3.1  (Test  Concept) 
to  TEXCOM  for  Cmt 

Tester 

ASAP 

T-285 

Final  TSP  to  Tester 

CBTDEV 

TNGDEV 

MATDEV 

ASAP 

T-270 

OTRR  #1 

TE)  OM 

T-270 

T-270 

Draft  TEP  Ch  3  to  TEXCOM 
for  Cmt 

Tee  '^r 

ASAP 

T-230 

Draft  TEP  Coord  with 

TIWG  Members  and  OEC 

Tester 

ASAP 

T-200 

Figure  3-6.  Milestones  for  Abbrev  Evaluate  TEP  Development 


Action 

1 

1 

Desired 

TIWG  Members  Return 

Coord  Draft  Cmts 

TIWG 

Members 

T-180 

Final  Draft  TEP  to 

TEXCOM  for  Approval 

Tester 

ASAP 

T-165 

TEP  Approved  by  TEXCOM 

TEXCOM 

ASAP 

T-150 

Draft  DTP  Complete 

Tester 

ASAP 

1 

VO 

o 

Strawman  TER  Complete 

Tester 

T-90 

OTRR  #2 

TEXCOM 

T-60 

T-60 

Final  DTP  Approval 

TEXCOM 

ASAP 

T-45 

Complete  Pilot  Test 

Tester 

T-3 

OTRR  #3 

TEXCOM 

T-2 

T-1 

Test  Start 

Tester 

T-Date 

Figure  3-6  (cont) .  Milestones  for  Abbrev  Evaluate  TEP  Development 


3-in 


STAFFING  LIST  FOR  DRAFT  AND  APPROVED  TEPs  FOR 
OTHER  THAN  FULL-EVALUATE  LEVEL  TEPs 


Number  of  copies  per  type  TEP 
Abbreviated 

evaluated  FDTE  CEP/CT 

Draft  Appr  Draft  Apor  Draft  Appr 

Deputy  Under  Secretary  of  the  Army 

(Operations  Research)  —  — 

Operational  Evaluation  Command  2  2  2*  2*  1*  1‘ 

Proponent  11  11  11 

TSM  111111 

Coproponent  11  11  11 

Project  Manager  11  1*  1* 

Materiel  Developer  11  1*  1*  11* 

USA  Air  Def  Artillery  School  1*  1*  1*  1*  1*  1* 

USA  Armor  School  1*  1*  1*  1*  1*  1* 

USA  Aviation  School  1*  1*  1*  1*  1*  1* 

USA  Chaplain  Center  and  School  1*  1*  1*  1*  1*  1* 

USA  Chemical  School  1*  1*  1*  1*  1*  1* 

USA  Field  Artillery  School  1*  1*  1*  1*  1*  1* 

USA  Engineer  School  1*  1*  1*  1*  1*  1* 

USA  Infantry  School  1*  1*  1*  1*  1*  1* 

USA  Army  Institute  of  Pers  and 

Resource  Mgt  1‘  1*  1*  1*  1‘  1* 

USA  Intelligence  Ctr  &  School  1*  1*  1*  1*  1*  1* 

USA  Intelligence  School  1*  1*  1*  1*  1*  1* 

USA  Military  Police  School  1*  1*  1*  1*  1*  1* 

USA  Missile/Munitions  Ctr/School  1*  1*  1*  1*  1*  1* 

USA  Ordnance  Ctr  and  School  1*  1*  1*  1*  1*  1* 

USA  Quartermaster  School  1*  1*  1*  1*  1*  1* 

USA  Signal  School  1*  1*  1*  1*  1*  1* 

USA  Transportation  School  1*  1*  1*  1*  1*  1* 

USASSC  and  Fort  Ben  Harrison  1*  1*  1*  1*  1*  1* 

USA  Combined  Arms  Center  and 

Fort  Leavenworth  1*  1*  1*  1*  1*  1* 

USA  Logistics  Center  1*  1*  1*  1*  1*  1* 

TEXCOM 

Methodology  and  Analysis  Dir  21  21  21 

Operations  Directorate_ 1  1* 1_ 1* 1  1* 


Figure  3-7.  Staffing  list  for  draft  and  approved  TEPs  for 
other  than  full-evaluate  level  TEPs 


Army  Research  Institute 
AMC  Liaison  Office 
Information  Systems  Dir 
Operations  Support  Dir 
Personnel,  Admin,  &  Log  Dir 
Resource  Management  Dir 

Test  directorate  or  HQ  TEXCOM  will  determine  if  staffing  is 
appropriate  for  each  specific  test. 


Figure  3-7.,  Staffing  list  for  draft  and  approved  TEPs  for 
other  than  full-evaluate  level  TEPs  (cont) 


1 

1 

!• 

1* 

1* 

1* 


1* 

1* 

!• 

1* 

1* 

1* 


1 

1 

1* 

1* 

1* 

1* 


1* 

1* 

1* 

1* 

1* 

1* 


1 

1 

1* 

1* 

1* 

1* 


1* 

1* 

1* 

1* 

1* 

1* 


CHAPTER  1.  INTRODUCTION  (lOE  responsibility).  The  format  for 
this  paragraph  will  be  the  same  as  for  a  regular  TEP. 

CHAPTER  2.  EVALUATION  PLAN  (lOE  responsibility).  The  format 
for  this  paragraph  will  be  the  same  as  for  a  regular  TEP.  In 
this  paragraph,  the  evaluator  must  provide  the  test  design 
concept  (paragraph  2.4  of  the  TEP)  to  guide  the  development  of 
the  combined  TT/OT  test  design. 

CHAPTER  3.  TEST  PLAN  (lOE  responsibility)  The  lOE  will  write 
a  brief  (single  paragraph)  narrative  of  the  combined  TT/OT 
plan  prepared  by  the  technical  tester  in  coordination  with  the 
operational  tester.  The  narrative  will  conclude  with  the 
annotation  that  a  copy  of  the  test  plan  is  at  a  designated 
appendix  to  the  TEP. 

APPENDIXES;  As  required. 


Figure  3-8.  TEP  format  for  a  combined  technical 
test/operational  test 


j 


CHAPTER  1.  INTRODUCTION  (lOE  responsibility) .  The  format  for 
this  paragraph  will  be  the  same  as  for  a  regular  TEP. 

CHAPTER  2.  EVALUATION  PLAN  (lOE  responsibility).  The  format 
for  this  paragraph  will  be  the  same  as  for  a  regular  TEP. 

There  will  be  no  paragraph  2.4  since  the  evaluation  plan  in 
this  TEP  will  not  be  the  basis  of  any  specific  test  design. 

CHAPTER  3.  TEST  PLAN  (lOE  responsibility).  The  lOE  will 
write  a  brief  (single  paragraph)  narrative  that  summarizes  the 
tests  outlined  as  data  sources  in  chapter  2.  Copies  of  test 
plans,  market  survey  or  investigation  plans,  and  modeling  or 
simulation  plans  may  be  attached  as  needed.  The  narrative 
will  conclude  with  the  annotation  that  a  copy  of  the  test 
plan(s)  is/are  at  designated  appendix (es)  to  the  TEP. 

APPENDIXES:  As  required. 


Figure  3-9.  TEP  format  for  an  operational  assessment  based 

on  other  than  a  user  test 


CHAPTER  1.  INTRODUCTION  (lOE  responsibility).  The  format  for 
this  paragraph  will  be  the  same  as  for  a  regular  TEP. 

CHAPTER  2.  EVALUATION  PLAN  (lOE  responsibility).  The  format 
for  this  paragraph  will  be  the  same  as  for  a  regular  TEP. 

CHAPTER  3.  TEST  PLAN-TEST  A  (tester  A  responsibility).  The 
format  for  this  paragraph  will  be  the  same  as  for  chapter  3  of 
a  regular  TEP. 

CHAPTER  4.  TEST  PLAN-TEST  B  (tester  B  responsibility).  The 
format  for  this  paragraph  will  be  the  same  as  for  chapter  3  of 
a  regular  TEP. 

APPENDIXES:  The  appendixes  will  be  the  same  as  for  a  regular 

TEP,  but  each  tester  must  prepare  tester  appendixes  for  this 
test.  Appendixes  will  be  redesignated  in  a  logical  fashion. 

NOTE:  Chapter  3  may  be  repeated  and  renumbered  accordingly 

for  multiple  tests. 


Figure  3-10.  TEP  format  for  multiple  user  tests 
supporting  a  single  evaluation 


MATERIEL  DEVELOPER  SUPPORT  PACKAGES 


System  support  package 

Statement  of  maintenance  procedures; 
draft  equipment  publications;  repair 
parts,  accessories;  special  and  common 
tools;  test  support  calibration  and 
maintenance/calibration  shop 
facilities;  and  personnel  skill 
requirements . 

New  equipment  training 

A  combination  of  documents,  training 
aids,  simulators,  programs  of 
instruction,  and  identification  of 
personnel  requirements  to  provide 
training  to  test  participants  and  to 
assist  the  trainer  in  developing  the 
system  raining  program. 

COMBAT  DEVELOPER  AND  TRAINER  SUPPORT  PACKAGES 


Doctrinal  and  organizational  TSP 


Means  of  employment 


Organization 


Logistic  concepts 


Draft  FMs;  doctrinal  employment  task 
lists;  statements  of  doctrine,  tactics, 
and  techniques. 

Statement  of  MOS,  basis  of  issue,  unit 
structures,  and  lines  of  command  or 
coordination  for  units  employing  the 
tested  system. 

Statement  of  applicable  supply, 
transportation,  and  maintenance 
concepts  and  procedures  for  the  tested 
system  that  is  compatible  with  the 
system  support  package  provided  by  the 
materiel  developer. 


Mission  profiles  Statement  of  types  of  combat  activities 

and  the  frequency  of  events 
(operational  mode  summary)  in  the 
combat  missions  in  the  form  of  either  a 
set  of  alternate  mission  profiles  or  a 
typical  profile,  plus  statistical 
distribution  of  frequency  of  events  and 
estimated  or  actual  duration  times  of 
events  and  time  between  events. 


Figure  3-11.  Test  support  packages 


Test  setting  Statement  of  plausible  situation  to 

show  interaction  among  threat,  friendly 
actions,  and  environment  involving  the 
tested  system. 

Threat  support  package 

Statement  of  the  potential  threat  in 
the  initial  operational  capability 
(IOC)  timeframe  related  to  the  tested 
system,  included  both  capabilities, 
typical  means  of  operating,  and  known 
methods  of  defeating  the  system. 

Training  support  package 

Conceptual  course  of  instruction, 
training  documents,  (individual 
training  plan,  individual/collective 
training  plan,  training  schedule,  task 
inventories,  ARTEPs,  soldier  manual, 
commander's  training  data  collection 
requirements) ,  and  personnel  selection 
criteria. 


Figure  3-11.  Test  support  packages  (cont) 
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Rgure  3-12.  Example  baseline  con’elation  matrix. 


REQUIREMEriTS  DOCUMENTS 


Figure  3-12  (cent).  Example  baseline  correlation  matrix. 


BCM  Requirements 

(1)  An  example  BCM  is  shown  in  figure  4-4.  This  greatly 
simplified  example  tracks  system  requirements  for  the 
hypothetical  "Cruising  Air  Launcher  (CAL)  Air  defense  System” 
over  four  requirements  documents  and  highlights  apparent 
discrepancies  in  values  and  wording  of  requirements.  Figure 
4-4  shows  several  different  types  of  discrepancies  that  the 
BCM  can  help  to  uncover. 

(a)  Requirement  1.1  tracks  through  O&O  plan,  the  system 
specifications,  and  the  ROC.  However,  the  MNS  says”...<  [less 
than)  20  enemy  planes”  and  the  other  documents  say”...<  [less 
than  or  equal  to]  20  enemy  planes.”  Identify  this  type  of 
discrepancy  to  the  MNS  author. 

(b)  Requirement  1.2  varies  in  the  number  of  seconds  from  5 
seconds  (in  the  O&O  and  MNS)  to  3  seconds  (in  the  system 
specifications  and  the  ROC) .  This  discrepancy  may  be 
justified,  or  it  may  be  an  error.  Pursue  this  kind  of 
discrepancy  until  it  is  resolved. 

(c)  Requirement  2.1  is  stated  in  miles  by  three  of  the 
documents  and  in  kilometers  by  the  fourth  (the  MNS) .  There 
would  be  no  discrepancy  if  the  MNS  required  ”3.2  kilometers,” 
because  3.2  kilometers  equals  2  miles.  When  the  numbers  are 
the  same  but  the  units  of  measure  are  different,  however,  the 
discrepancy  must  be  resolved. 

(d)  Requirement  2.1  does  not  track  because  the  opening  phrase 
"Given  target  detection”  is  left  out  of  the  ORD  requirement. 
This  omission  could  be  either  an  oversight  or  intentional.  If 
the  omission  is  intentional,  the  meaning  changes  entirely. 
Follow  up  and  resolve  the  discrepancy. 

(e)  Requirement  3.2  does  not  appear  in  the  ORD  column, 
indicating  that  the  requirement  has  disappeared  in  the  course 
of  the  program.  Follow  up  on  why  requirement  3.2  is  no  longer 
necessary,  or  ensure  that  it  is  otherwise  accommodated. 

(2)  The  example  BCM  in  figure  4-2  shows  sample  I&C 
corresponding  to  the  system  requirements  and  the  logical  MOEs. 
These  MOEs  seem  reasonable;  however,  one  possible  type  of 
discrepancy  that  might  typically  occur  is  shown. 


Figure  3-13.  BCM  requirements 


(a)  The  discrepancy  occurs  between  the  probability  of  kill 
(Pk)  requirement  1.1  and  the  probability  of  detection  (Pd) 
requirement  2.1.  The  MOE  for  1.1  is  defined  as  Pk  where  Pk  = 
#K  /  T#TGTS;  and  #K  =  the  number  of  enemy  planes  killed  by  CAL 
in  a  given  battle  sequence;  and  T#TGTS  =  the  total  number  of 
possible  targets  in  a  given  battle  sequence.  The  MOE  for  2.1 
is  defined  as  Pd  where  Pd  =  #D  /  T#TGTS;  and  #D  =  the  number 
of  planes  detected  by  CAL  in  a  given  battle  sequence;  and 
T#TGTS  is  the  same  total  number  of  possible  targets. 

(b)  Obviously,  when  Pk  and  Pd  are  defined  in  this  manner,  Pk 
must  be  less  than  or  equal  to  Pd.  However,  the  requirements 
documents  state  Pk  >  .96,  and  Pd  >  .91,  so  the  evaluator  must 
resolve  the  discrepancy.  The  Pk  referred  to  is  probably  the 
Pk  given  a  detection.  The  evaluator  must  ensure  that  this  is 
in  fact  the  requirement,  and  then  the  MOE  is  more  specifically 
stated. 


Figure  3-13.  BCM  requirements  (cont) 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  1 — INTRODUCTION 

CHAPTER  1.  INTRODUCTION.  This  chapter  provides  necessary 
information  to  understand  the  bases  of  both  the  test  and  of 
the  evaluation.  This  chapter  is  prepared  by  either  the 
evaluator  or  the  tester  depending  on  the  type  of  test  or 
evaluation.  See  figure  3-3  for  allocation  of  responsibilities 
for  the  preparation  of  this  chapter. 

1.1.  PURPOSE.  State  the  purpose  for  conducting  the 
evaluation  and  for  planning  the  testing.  State  any  decision 
supported  by  the  evaluation  and  form  an  extent  of  evaluation 
or  assessment.  Identify  all  testing  outlined  in  chapters  2 
and  3  of  the  TEP. 

1.2.  SCOPE.  Address  the  breadth  of  the  data  sources  to  be 
used  to  prepare  the  test  and  evaluation/assessment  reports. 
Provide  an  overview  relevant  models,  analyses,  tests, 
equipment,  and  other  resources  which  together  are  the  basis 
for  any  planned  evaluation  or  assessment. 

1.3.  BACKGROUND.  Include  the  background  of  the  system 
development  and  T&E  of  the  system.  If  the  details  are  so 
voluminous  that  continuity  of  the  TEP  is  lost,  they  should  be 
summarized  here  and  stated  in  detail  in  appendix  B.  Appendix 
B  is  optional  if  the  details  can  be  adequately  covered  in  this 
paragraph. 

1.3.1.  PROGRAM  BACKGROUND.  Include  an  overview  of  the 
program,  its  acquisition  strategy,  the  system's  anticipated 
use  to  the  Army,  and  the  deficiency  identified  in  the  Mission 
Area  Analysis  (MAA)  that  the  system  is  to  correct.  Identify 
the  next  program  decision  to  be  supported  by  the  testing  and 
evaluation  and  the  decision  to  be  made. 

1.3.2.  TEST  AND  EVALUATION  BACKGROUND.  Include  a  summary  of 
all  OTE,  DTE,  and  contractor  test  to  date.  Both  the  scope  of 
T&E  and  the  results  of  the  T&E  are  to  be  included. 

1.3.3.  COEA/OTE  RELATIONSHIP.  Required  for  any  system  for 
which  a  COEA  is  done  and  OTE  is  conducted.  Describe  the 
linkage  between  the  COEA  and  the  planned  results  of  OTE. 


Figure  3-14.  Detailed  guidance  for  the  development  and 
documentation  of  Test  and  Evaluation  Plans,  Chapter  1 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  1 — INTRODUCTION  (cont) 

1  4  SYSTEM  DESCRIPTION.  Include  a  description  of  the  system 
(or  combat/training  development  product) .  Describe  the  system 
and  its  major  roles,  missions,  and  components  or 
characteristics.  If  more  detail  is  required  to  adequately 
understand  the  system,  a  lengthy  system  description  can  be 
included  as  appendix  C.  Appendix  C  is  optional  if  the  details 
can  be  adequately  covered  in  this  paragraph. 

1.5.  PROJECTED  THREAT.  Define  the  approved  threat  in  the 
post-IOC  timeframe  of  the  tested  system  and  include 
capabilities,  typical  means  of  operation,  and  known  methods  of 
defeating  the  system.  If  more  detail  is  required  to 
adequately  understand  the  threat,  a  lengthy  threat  statement 
can  be  included  as  appendix  D.  Appendix  D  is  optional  if  the 
details  can  be  adequately  covered  in  this  paragraph. 

1.6.  TEST  AND  EVALUATION  MILESTONES.  List  all  the  milestones 
important  to  success  or  completion  of  T&E  leading  to  a 
milestone  decision  review  or  other  event  supported  by  the  T&E 
effort.  The  list  may  be  incorporated  as  a  part  of  the 
paragraph  or  may  be  called  out  in  the  paragraph  and  included 
as  a  table  or  a  figure  in  the  TEP. 

1.7.  OPERATIONAL  ISSUES  AND  CRITERIA  (OIC) .  List  the  OIC 
used  to  evaluate  the  system,,  construct  an  evaluation  plan,  and 
develop  a  test  design  for  the  system  (or  combat/training 
developments  product).  For  materiel  and  IMA  systems,  break 
the  OIC  down  into  COIC  (prepared  by  the  CBTDEV/functional 
proponent  and  AOIC  prepared  by  the  evaluator. 

CBTDEV  or  TNGDEV  products,  only  OIC  developed  by  TRADOC  are 
required.  COIC  and  AOIC  collectively  constitute  the  OIC  for 
the  TEP.  Each  issue  must  have  at  least  one  associated 
criterion.  State  each  OIC  set  in  full  to  irclude  issue, 
scope,  criteria,  and  rationale.  The  evalue  may  add 

additional  evaluator  criteria  to  a  critica^  sue.  Criteria 

which  are  a  part  of  the  original  COIC  set  w  always  have 
both  a  measure  (parameter)  and  a  threshold  (Vvlue).  Criteria 
developed  by  the  evaluator  may  have  only  a  measure.  In  those 
cases  where  the  evaluator  criterion  lacks  a  threshold  (value), 
the  criterion  will  be  labeled  ••  investigative"  and  only  the 
measure  (parameter)  specified. 


Fiaure  3-14.  Detailed  guidance  for  the  development  and 

documentation  of  Test  and  Evaluation  Plans,  Chapter  1  (cont) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  1 — INTRODUCTION  (cont) 

1.7.1.  CRITICAL  OPERATIONAL  ISSUES  AND  CRITERIA  (COIC) .  This 
paragraph  contains  the  approved  COIC  which  are  directly 
extracted  from  the  approved  TEMP. 

1.7.2.  ADDITIONAL  OPERATIONAL  ISSUES  AND  CRITERIA  (AOIC) . 

This  paragraph  contains  AOIC  developed  by  the  evaluator  to 
complement  and  supplement  approved  COIC  to  completely  address 
all  aspects  of  system  operational  effectiveness  and 
suitability. 

1.7.3.  BASELINE  CORRELATION  MATRIX  (BCM) .  This  paragraph 
calls  out  the  BCM  which  is  then  included  as  a  figure  or  table 
in  the  TEP  on  a  facing  page.  A  BCM  is  only  required  for  full 
evaluation  TEP.  Instructions  for  the  preparation  of  the  BCM 
are  contained  in  chapter  3. 


Figure  3-14.  Detailed  guidance  for  the  development  and 
documentation  of  Test  and  Evaluation  Plans,  Chapter  1  (cont) 


SAMPLE  ISSUE  AND  CRITERIA  SET 

Issue ♦  Is  the  ***  system  effective  at  determining 
prioritized  target  information  to  support  ***  in  the  close 
support  role? 

Scope.  This  issue  addresses  the  speed  and  accuracy  with 
which  the  ***  system  can  search,  detect,  and  locate  heat 
emitting  targets  in  the  European  theater.  The  probability  of 
detecting  a  target  will  be  examined  based  on  the  type  target, 
IR  cross  section,  ***  system  speed,  search  pattern  and  target 
density. 

Criterion.  The  ***  system  must  have  a  90%  chance  of 
detecting  threat  vehicular  targets  within  2  minutes  and 
locating  them  with  a  25  meter  CEP  accuracy. 

Rationale.  ******** 


Figure  3-15.  Sample  issue  and  criteria  set 


Figure  3-16.  Example  location  function 
dendritic. 
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Figure  3-1 6  (cent).  Example  location  function 
dendritic. 
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Figure  3-1 6  (cent).  Example  location  function 
dendritic. 


FACTORS  AND  CONDITIONS 


FACTORS 

CONTROL 

Range  of  engagement 

Systematically  varied 

Light  conditions 
Target  movement 

Systematically  varied 
Systematically  varied 

Threat  arrays 

Systematically  varied 

NBC 

Systematically  varied 

Terrain  (phase  I) 
Terrain  (phase  II) 
Enemy  action 

Systematically  varied 
Tactically  varied 
Systematically  varied 

Battlefield 

obscuration 

EW  environment 

Systematically  varied 

Systematically  varied 

Personnel 

Held  constant 

Organization 
Doctrine/ tactics 
Logistics  support 
Communications 
status 

Enemy  target 

Held  constant 

Held  constant 

Held  constant 
Tactically  varied 

Tactically  varied 

Weather 

Uncontrolled 

System  operating 
status 

Uncontrolled 

CONDITIONS 

100-500ni, 
501-900in, 
900-1300in 
Day,  night 
Moving , 
stationary 
lAW  Threat 
Spt  Package 
No  MOPP, 

MOPP  2,  MOPP  4 
Flat,  rolling 
Rugged ,  swamp 
Attack, 
defend 
No  smoke, 
smoke 

lAW  threat 
spt  package 
5th-95th 
percentile 
Battery  level 
lAW  D&O  TSP 
ORG,  DS 
Radio-voice, 
radio-digital 
Troops , 
vehicle, 
bunker 
Rain,  dry, 
snow 

Fully  opnl, 
degrade  mob, 
degrade  fipwr 
nonopn3 


Figure  3-17.  List  of  typical  factors  and  conditio;” 
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Rgure  3-18.  Example  event  dendritic, 


EVENT  DENDRmC 


Rgure  3-18  (cont).  Example  event  dendritic. 
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Figure  3-1 9.  Example  mission  performance  dendritic. 
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Figure  3-19  (cent).  Example  mission  performance  dendritic. 


Figure  3-19  (cont).  Example  mission  performance  dendritic 
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e  3-19  (cont).  Example  mission  performance  dendritic. 


Figure  3-20.  Data  source  matrix 


LEVEL 

DESCRIPTION 

POSSIBLE  FORMS 

EXAMPLES  OF  CONTENT 

DISPOSITION 

Level  1 

Data  in  their 

Complete  data 

1. 

All  reported  target 

Accinulated 

data  "raw 

original  form. 

collection  sheets, 

presentations  and 

during  trials 

data" 

Results  of  field 

exposed  camera  film, 

detection. 

for  processing. 

trials  just  as 

voice  recording 

2. 

Clock  times  of  all 

Usually 

recorded. 

tapes,  original 

events. 

discarded  after 

instrimentation 

3. 

Azimuth  and  vertical 

use.  Not 

magnetic  tape  or 

angle  from  each 

ordinari ly 

printouts,  original 

flash  base  for  each 

given  to 

videotapes,  filled 

flash. 

another  agency. 

questionnaires, 

4. 

Recording  tapes  of 

Wot  published. 

interview  notes. 

interviews. 

Level  2 

Data  taken  from 

Confirmed  and 

1. 

Record  of  all  valid 

Produced  during 

data 

the  raw  form  and 

corrected  data 

detections. 

processing. 

"reduced 

consol idated. 

collection  sheets. 

2. 

Start  and  stop  times 

Usual ly 

data" 

Invalid  or 

film  with  extraneous 

of  all  applicable 

discarded  after 

unnecessary  data 

footage  deleted, 

events. 

use.  Not 

points  deleted. 

corrected  tapes  of 

3. 

Computed  impact 

published. 

Trials  declared 

printouts,  and 

points  of  each  round 

"no  test" 

original  raw  data 

flashed. 

deleted. 

with  "no  test"  events 

4. 

Confirmed  interview 

marked  out. 

records. 

Level  3 

Data  which  have 

Spread  sheets. 

1. 

Counts  of  detections 

Not  usually 

data 

been  checked  for 

tables,  typed  lists. 

arranged  in  sets 

published  but 

"ordered 

accuracy  and 

ordered  and  labeled 

showing  conditions 

made  available 

data" 

arranged  in 

printouts,  purified 

under  which 

to  analysts. 

convenient  order 

and  ordered  tape. 

detections  occurred. 

Usually  stored 

for  handling. 

edited  film,  edited 

2. 

Elapsed  times  by 

in 

Operations 

magnetic  tap>es. 

type  events. 

institutional 

limited  to 

ordered  punch  cards. 

3. 

Impact  points  of 

data  banks. 

counting  and 

rounds  by  condition 

All  or  part  may 

elementary 

under  which  fired. 

be  published  as 

arithmetic. 

4. 

Interview  comments 

supplements  to 

categorized  by  type. 

test  report. 

Level  4 

Data  which  have 

Tables  or  graphs 

1. 

Percentage  of 

Published  as 

data 

been  sutmarized 

showing  totals. 

presentations 

the  basic 

"findings" 

by  elementary 

means,  medians. 

detected. 

factual 

or 

mathematical 

modes,  maximuns, 

2. 

Mean  elapsed  times. 

findings  of 

"sunnary 

operations. 

minimums,  quartiles. 

3. 

Calculated  probable 

test  report. 

statis¬ 

Operations 

deciles,  percentiles. 

errors  about  the 

tics" 

limited  to 

curves,  or  standard 

centers  of  impact  or 

descriptive  sum¬ 

deviations. 

conditions. 

maries;  no 

Qualitative  data  in 

4. 

Bar  graph  showing 

judgments  or 

form  of  lists. 

relative  frequency 

inferences.  Does 

histographs,  counts 

of  each  category  of 

not  go  beyond 

by  type,  or  sunnary 

comment. 

what  was  observed 

statements. 

in  test. 

Figure  3-21.  Levels  of  data 
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LEVEL 

DESCRIPTION 

POSSIBLE  FORMS 

EXAHPLES  OF  CONTENT 

DISPOSITION 

Level  5 

Data  resulting 

Results  of  primary 

1.  Inferred  probability 

Published  in 

data 

from  statistical 

statistical 

of  detection  with 

evaluation 

“analysis” 

tests  of 

techniques  such  as 

its  confidence 

reports. 

or  “infer- 

hypothesis  or 

T-tests,  Chi-square, 
F-test,  analysis  of 

interval . 

(If  evaluation 

ential 

interval 

2.  Significance  of 

statis- 

estimation. 

variance,  regression 

difference  between 

report  is  part 

tics" 

Execution  of 

analysis,  contingency 

two  mean  elapsed 

of  test  report, 

planned  analysis 

table  analyses  and 

times. 

the  level  5 

data.  Includes 

other  associated 

3.  Significance  of 

analysis 

both  comparisons 

confidence  levels. 

difference  between 

results  are 

and  statistical 

Follow-on  tests  of 

observed  probable 

presented 

significance 

hypotheses  arising 

error  and  criterion 

separately  from 

level .  Judgments 

from  results  of 

threshold. 

the  level  4 

limited  to 
analysts 
selection  of 
techniques  and 
significant 
levels. 

earlier  analysis,  or 
fallback  to  alternate 
nonparametric 
technique  when 
distribution  of  data 
does  not  support 
assumption  of 
normality. 

Qualitative  data  in 
the  form  of 
prevailing  consensus. 

4.  Magnitude  of 

difference  between 
categories  of 
comments. 

findings.) 

Level  6 

Data  resulting 

Insertion  of  test 

1.  Computation  of 

Published  as 

data 

from  further 

data  into  a 

probability  of  hit 

appropriate  in 

“extended 

analytic 

computational  model 

based  on  target 

evaluation 

analysis" 

treatment  going 

or  a  combat  simu- 

detection  data  from 

reports. 

or 

beyond  primary 

1 at  ion,  aggregation 

test  combined  with 

operations 

statistical 
analysis, 
combination  of 
analytic  results 
from  different 
sources,  or 
exercise  of 
simulation  or 
models. 

Judgments  limited 
to  analysts' 
choices  only. 

of  data  from 
different  sources 
observing  required 
disciplines,  curve 
fitting  and  other 
analytic 

generalization,  or 
other  operations 
research  techniques 
such  as  application 
of  queuing  theory, 
inventory  theory, 
cost  analysis,  or 
decision  analysis 
techniques. 

separate  data  or 
probability  of  hit 
given  a  detection. 

2.  Exercise  of 
attrition  model 
using  empirical  test 
times  distribution. 

3.  Determination  of 
whether  a  trend  can 
be  identified  from 
correlation  of  flash 
base  accuracy  data 
under  stated 
conditions  from 
different  sources. 

4.  Delphi  technique 
treatment  of 
consensus  of 
interview  comments. 

Level  7 

Data  conclusions 

Stated  conclusions  as 

1.  Conclusion  as  to 

Published  as 

data 

resulting  from 

to  issues,  position 

whether  probability 

the  basic 

“conclu¬ 

applying 

statements. 

of  detection  is 

evaluative 

sions"  or 

evaluative 

challenges  to 

adequate. 

conclusions  of 

evaluation 

military 
judgments  to 
analytic  results. 

validity  or  analysis. 

2.  Conclusion  as  to 
timeliness  of  system 
performance. 

3.  Conclusion  as  to 
military  value  of 
flash  base  accuracy. 

4.  Conclusion  as  to 
main  problems 
identified  by 
interviewees. 

evaluation 

reports. 

Figure  3-21. Levels  of  data  (cont) 
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A.  TRIAL  HEADER  RECORD 

•  WHEN,  WHAT,  WHO,  HOW  INSTRUMENTED 


B.  INTERVISIBILITY  SEGMENT  RECORDS 

•  WHEN  DID  PAIRS  OF  PLAYERS  HAVE  "INTERVISIBILITY" 

•  COULD  EITHER  PLAYER  BE  REASONABLY  EXPECTED  TO 
ENGAGE  THE  OTHER 

•  DID  THE  TEST  SYSTEM  DETECT,  DISPLAY,  IFF 


C.  TIME  LINE  RECORDS 

•  WHEN  AND  WHERE  DID  EVENTS  IN  ENGAGEMENT 
SEQUENCES  OCCUR 


D.  FIRING  RECORDS 

•  WHEN  DID  WEAPON  FIRINGS  OCCUR 

•  UNDER  WHAT  CONDITIONS 

•  WHAT  PR’s  ASSOCIATED  WITH  FIRING  CONDITIONS 

•  WHATW/  SIMULATED  RESULT 


E.  MILES  DEATH  RECORDS 

•  WHICH  PLAYERS  PURPORTEDLY  KILLED  IN  MILES 
ENGAGEMENTS 


SYSTEM  SEARCH  FILE  LOADING  RECORDS 
•  ALL  SORTS  OF  DETAILED  SYSTEM  DATA 


GENERAL  RECORD  TYPES 

A.  TRIAL  HEADER  (TH) . 

1.  One  per  trial. 

2.  Specifies  trial  conditions.  .  .  *  -,1 

3.  Specifies  composition  of  red  and  blue  forces  during  trial. 

B  INTERVISIBILITY  SEGMENT  (IS)  (POTENTIAL  ENGAGEMENT  OPPORTUNITY) . 

1  one  record  for  each  t'ce  period  during  a  trial  when  actions  could 

P0t|"tiJj]{J^°fjj5  general  parameters  applicable  to  this  IS  record 

(parameters  vary  depending  on  pair). 

3.  Identifies  for  each  player: 

a.  Whether  an  engagement  opportunity  occurred. 

b.  How  many  engagements  occurred,  if  any. 

^  if'^^Zero^oLl^^or  more^than  one  record  associated  with  each  specific  IS 

2.  Applicable  only  to  partial  or  completed  engagements.  ^  ^ 

3  Categorizes  each  STL  as  a  'reduction  time  or  a  servicing  time. 

4!  Specifies  time,  ranges,  and  other  applicable  parameters  for  each 

^‘'’l."ldo“i'f°el"tho  number  of  bursts  corresponding  to  the  firing  period 
(if  any). 

^  ^  1  ^ '^Record ^^as locution  varies  depending  on  players  involved  and  which 

player  multiple  records,  associated  with  ^TL  (if  any). 

b  Zero,  one,  or  multiple  records  associated  with  JS  (if  no  STL), 

c*.  Firings  which  do  not  result  in  pairings  are  included. 

2  Identifies  shooter,  target  weapon,  time,  and  all  available  relevant 
firing  parameters  for  each  shot  fired  or  missile  launched  (includes  gaggle 
ID  as  appropriate). 

E  AUXILIARY  FIRING  REPORT  (AFR).  ,  . 

1  R^rts  firings  indicated  by  MILES  or  video  tape  analysis,  one 

record  Iq  weapon  type  and  target  ID  as  well  as 

approximate  time  of  death. 

^3.  Gives  results  of  firing  (kill,  near-mess,  unknown). 

i^^^H\'rtfpS\%^cotdSr  «ch'^^'^c  unit  every  1.6  seconds  during  trial 
2!  Gives  record  of  search  file  loading  by  search  file  number,  including 
object  classification  code,  actual  target  ID  (if  correlated),  priority 
?oin?L  and  system  actions  against  that  search  file  during  previous 
1.6  seconds. 

G.  Pk  TABLE  (PKT)- -PROVIDES  RTCA  Pk  TABLES,  OR  MODIFICATIONS,  IN  MACHINE 
READABLE  FORMAT. _ _ _ _ _ _ _ _ _ 

Figure  3-23.  Outline  of  data  base  required  for  analysis. 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2— EVALUATION  PLAN 

CHAPTER  2 .  EVALUATION  PLAN  (when  this  chapter  is  prepared  by 
the  tester,  it  is  called  the  ANALYSIS  PLAN).  This  chapter  is 
prepared  by  either  the  evaluator  or  the  tester  depending  on 
the  type  of  test  or  evaluation.  When  the  evaluator  prepares 
the  chapter,  it  is  called  the  Evaluation  Plan.  When  the 
tester  prepares  the  chapter,  it  is  called  the  Analysis  Plan. 
Paragraph  side  headings  will  use  the  evaluation  terminology  if 
prepared  by  the  evaluator  and  analysis  terminology  if  prepared 
by  the  tester.  When  the  tester  prepares  an  analysis  plan  in 
this  chapter,  the  effort  is  simplified  and  abbreviated. 

2.1.  EVALUATION  APPROACH  (tester  uses  ANALYTIC  APPROACH). 

2.1.1.  OVERVIEW  OF  THE  EVALUATION/ANALYTIC  APPROACH.  This 
paragraph  summarizes  the  overall  approach  to  the  evaluation  of 
the  system. 

2.1.2.  AGGREGATION  OF  OPERATIONAL  EFFECTIVENESS.  This 
paragraph  shows  the  methodology  by  which  the  answers  to  the 
issue  questions  will  be  further  analyzed  and  consolidated  to 
permit  conclusions  or  predictions/assessments  to  be  drawn  on 
the  overall  operational  effectiveness  of  the  system. 

2.1.3.  AGGREGATION  OF  OPERATIONAL  SUITABILITY.  This 
paragraph  shows  the  methodology  by  which  the  answers  to  the 
issue  questions  will  be  further  analyzed  and  consolidated  to 
permit  conclusions  or  predictions/assessments  to  be  drawn  on 
the  overall  operational  suitability  of  the  system. 

2.2.  OPERATIONAL  ISSUES.  This  paragraph  lists  in  turn  each 
of  the  operational  issues  and  associated  criteria  stated  in 
TEP  paragraph  1.7. 

2.1.  ISSUE  1.  This  paragraph  contains  the  issue  statement 
taken  in  turn  from  the  issue  statements  in  paragraph  1.7.  The 
issues  are  stated  in  the  form  of  a  question  for  which  a 
response  will  make  a  significant  contribution  to  the  decision 
making  process.  Scope  and  rationale  need  not  be  restated  here 
as  they  are  already  provided  in  paragraph  1.7  and  restatement 
would  be  unnecessary  duplication. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
Documentation  of  Test  and  Evaluation  Plans,  Chapter  2 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2 — EVALUATION  PLAN  (cont) 


2. 2. 1.1.  CRITERION  1-1.  This  paragraph  contains  the 
criterion  statement  taken  in  turn  from  the  criteria  statements 
for  the  issues  in  paragraph  1.7.  The  evaluator  may  add 
additional  evaluator  criteria  to  a  critical  issue.  Criteria 
which  are  a  part  of  the  original  COIC  set  will  always  have 
both  a  measure  (parameter)  and  a  threshold  (value) .  Criteria 
developed  by  the  evaluator  may  have  only  a  measure.  In  those 
cases  where  the  evaluator  criterion  lacks  a  threshold  (value) , 
the  criterion  will  be  labeled  "investigative"  and  only  the 
measure  (parameter)  specified. 

2. 2. 1.1.1.  MEASURE  OF  EFFECTIVENESS  (MOE) /MEASURES  OF 
PERFORMANCE  (MOP)  1-1-1.  Derive  and  state  all  MOE/MOP 
associated  with  the  criterion.  Each  criterion  must  have  one 
or  more  MOE  or  MOP.  When  the  tester  prepares  chapter  2,  there 
will  normally  be  only  MOP.  The  evaluator  may  find  it 
necessary  to  use  both  MOE  and  MOP.  This  paragraph  must  define 
the  individual  MOE/MOP. 

a.  ANALYTIC  DESIGN.  The  analytic  design  is  a  description  of 
the  logical  process  to  be  used  to  measure  the  MOE  or  MOP.  The 
MOE  and  MOP  may  be  grouped  by  like  analytic  designs. 

b.  ANALYTIC  PROCEDURES.  Analytic  procedures  are  a 
description  of  the  anticipated  framework  within  which  the  data 
will  be  analyzed.  This  section  serves  as  a  "road  map"  for  the 
analyses  which  are  intended  to  identify  or  support  evaluative 
conclusions.  Parametric  and  nonparametric  technigues  should 
be  specified,  assumptions  should  be  discussed,  and  statistical 
software  packages  should  be  specified.  Group  the  MOE  and  MOP 
by  like  analysis  concepts. 

c.  DATA  PRESENTATIONS.  The  analyst  proposes  the  data 
displays,  tables,  figures,  or  other  forms  of  presentation 
appropriate  for  displaying  data  in  reports.  Tables  must  be 
provided  for  cell  counts,  percentages,  measures  of  central 
tendency  and  variability,  and  for  ANOVA  tables.  Group  MOE/MOP 
by  like  presentations. 

2. 2. 1.1. 2.  MOE/MOP  1-1-2. 

through 

2.2.1.1.x.  MOE/MOP  1-1-X. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
Documentation  of  Test  and  Evaluation  Plans,  Chapter  2  (cont) 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2— EVALUATION  PLAN  (cont) 

2.2.1.1. X+1.  CRITERION  ANALYTIC  SUMMARY.  This  summary 
describes  the  logical  process  which  the  analyst  intends  to  use 
to  resolve  the  criterion.  Group  the  MOE  and  MOP  by  like 
analytic  designs  and  concepts  and  describe  how  the  data  from 
the  MOE  and  MOP  sources  will  be  integrated. 

2 . 2 . 1 . 2 .  CRITERION  1-2 . 
through 

2.2. 1. y.  CRITERION  1-y. 

2.2.1. y+l.  ISSUE  ANALYTIC  SUMMARY.  This  summary  describes 
the  logical  process  which  the  analyst  intends  to  use  to 
resolve  the  issue.  Group  analyses  by  like  analytic  designs 
and  concepts  and  describe  how  the  data  from  the  criterion 
summaries  will  be  integrated.  The  plan  must  call  for  a  direct 
answer  to  the  question  asked  in  the  issue  statement  for  the 
final  report. 

2.2.2.  ISSUE  2. 
through 

2.2. n.  ISSUE  n. 

(NOTE:  THIS  IS  THE  END  TESTER  ANALYSIS  PLANS) 

2.3.  DATA  SOURCE  MATRIX  (DSM) .  All  sources  of  data  to  be 
used  in  an  evaluation  or  assessment  must  be  described  in  the 
TEP.  Each  MOE  or  MOP  must  be  addressed  by  at  least  a  primary 
data  source.  Any  test,  model /simulation,  market 

survey/ investigation,  or  study/analysis  listed  as  a  p'  nary  or 
secondary  data  source  for  any  MOP  must  be  described 
paragraphs  2.4  or  2.5.  The  matrix  cells  are  to  be  fs  u  with 
"P”  or  to  indicate  whether  the  data  source  will  pi  vide  a 
’•primary”  or  "secondary”  contribution  to  the  evaluation. 

Matrix  cells  are  left  blank  for  data  sources  which  are  neither 
primary  or  secondary.  Each  MOP  must  have  at  least  one  primary 
data  source. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2 — EVALUATION  PLAN  (cont) 

2.4.  USER  TEST  APPROACH  FOR  THE  TEST  NAME  (E.G.  lOT,  FOT, 

EUT,  FDT) .  Use  this  paragraph  to  describe  a  user  test  for 
which  the  evaluator  has  derived  the  test  approach  and  concept 
and  whose  test  design  is  described  in  chapter  3.  Repeat  this 
paragraph  as  necessary  (paragraph  2.4,  2.5,  etc.)  for  multiple 
tests. 

2.4.1.  TEST  SCOPE.  The  scope  identifies  the  types  of  test 
scenarios  and  test  events  that  are  required  to  address  the 
operational  test  MOPs  (OTMOPs) . 

2.4.2.  FACTORS  AND  CONDITIONS.  Factors  are  the  test 
variables  identified  as  likely  to  affect  test  event  outcome. 
Conditions  are  the  discrete  values  that  factors  assume  or  are 
expected  to  assume. 

2.4.3.  TEST  DESIGN  MATRIX (ES) .  Using  test  events  and 
scenarios  identified  in  the  scope  and  stated  factors  and 
conditions,  the  evaluator  develops  matrix (es)  needed  for 
grouping  combinations  of  test  conditions  into  ••trials,” 
"missions,”  or  "phases.”  Matrixes  are  formed  by  crossing 
factors  and  conditions  with  one  another.  Large  matrixes 
should  be  reduced  to  several  smaller  ones.  Normally  the 
analytic  designs  described  in  paragraphs  2 . 1  and  2 . 2  will 
reflect  these  matrixes. 

2.4.4.  SAMPLE  SIZES.  Based  on  the  test  design  matrix (es) , 
the  evaluator  estimates  the  aggregate  sample  sizes  likely  to 
occur  under  each  combination  of  factors  and  conditions  for 
tests  of  various  lengths  and  for  alternative  formulations  of 
test  elements.  Once  sample  sizes  are  determined,  they  are 
tabulated  as  matrixes  showing  numbers  of  valid  test  events  to 
be  conducted  under  various  combinations  of  test  conditions. 

2.4.5.  REQUIREMENTS  FOR  A  DATA  AUTHENTICATION  GROUP  (DAG). 

The  evaluator  identifies  whether  or  not  there  are  requirements 
for  a  DAG.  This  paragraph  outlines  the  role  of  the  DAG  in 
data  validation  and  in  engineering  analysis  of  test  data. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
Documentation  of  Test  and  Evaluation  Plans,  Chapter  2  (cont) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2— EVALUATION  PLAN  (cont) 

2.5.  OTHER  SOURCES  OF  DATA.  All  sources  Of  data  to  be  used 
in  an  evaluation  or  assessment  must  be  described  in  the  TEP. 
Any  test,  model,  simulation,  market  survey,  market 
investigation,  study,  or  analysis  listed  as  a  primary  or 
secondary  data  source  for  any  MOE/MOP  must  be  described  in 
this  paragraph.  Each  separate  test  and  other  effort 
comprising  a  discrete  data  source  will  have  its  own 
subparagrapli  in  paragraph  2.5  and  the  paragraph  numbering  will 
be  adjusted  accordingly  (e.g. ,  2.5.1.  FDTE,  2.5.2.  TTl, 
2.5.3.  TT2,  2.5.4.  Contractor  Test,  2.5.5.  COEA  Model, 
2.5.6.  Other  Model,  2.5.7.  Simulation,  2.5.8.  Market 
Survey) .  This  paragraph  is  omitted  when  the  tester  prepares 
chapter  2  of  the  TEP. 

2.5.1.  USER  TEST  NAME  (e.g.,  FDT,  FDE,  CEP,  CT) .  Use  this 
paragraph  to  describe  a  user  test  which  will  be  used  in  the 
evaluation  and  for  which  the  evaluator  has  not  derived  the 
test  approach  and  concept  and  whose  test  design  is  described 
in  other  documents.  Repeat  this  paragraph  as  necessary  for 
multiple  tests.  This  paragraph  varies  in  scope  and  complexity 
depending  on  the  size  and  sophistication  of  the  anticipated 
test.  It  provides  the  basis  for  understanding  the  value  and 
context  of  the  data  derived  from  this  source. 

2. 5. 1.1.  TEST  SCOPE.  Derive  the  scope  from  the  plan  of  the 
test  in  sufficient  detail  to  provide  understanding  of  the 
test,  the  MOP  to  be  addressed,  test  events,  and  type  of  data 
collected. 

2. 5. 1.2.  FACTORS  AND  CONDITIONS.  Factors  and  conditions  from 
the  test  plan  are  described  in  sufficient  detail  to  provide  a 
context  for  the  data  to  be  use 

2.5.2.  DEVELOPMENT  TEST  NAME  g. ,  DT,  Contractor  Test, 
etc.).  Use  this  paragraph  to  escribe  a  DT,  a  contractor 
test,’  or  any  other  test,  demonstration,  or  review.  The  test 
design  will  be  described  in  the  test  plan  for  the  test. 

Repeat  this  paragraph  as  necessary  for  multiple  tests.  This 
paragraph  varies  in  scope  and  complexity  depending  on  the  size 
and  sophistication  of  the  anticipated  test.  It  provides  the 
basis  for  understanding  the  value  and  context  of  the  data 
derived  from  this  source. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
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) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2— EVALUATION  PLAN  (cont) 

2. 5. 2.1.  TEST  SCOPE.  Derive  the  scope  from  the  plan  of  the 
test  in  sufficient  detail  to  provide  understanding  of  the 
test,  the  MOP  to  be  addressed,  test  events,  and  type  of  data 
collected. 

2. 5. 2. 2.  FACTORS  AND  CONDITIONS.  Factors  and  conditions  from 
the  test  plan  are  described  in  sufficient  detail  to  provide  a 
context  for  the  data  to  be  used. 

2.5.3.  MODEL  OR  SIMULATION  NAME.  Use  this  paragraph  to 
describe  a  model  or  simulation  which  will  provide  data  for  the 
evaluation.  Repeat  this  paragraph  as  necessary  for  multiple 
models  and  simulations.  This  paragraph  varies  in  scope  and 
complexity  depending  on  the  size  and  sophistication  of  the 
anticipated  model  or  simulation.  It  provides  the  basis  for 
understanding  the  value  and  context  of  the  data  derived  from 
this  source.  Validation,  verification,  and  accreditation 
sources  and  dates  for  the  model  or  simulation  must  be  listed 
in  this  paragraph. 

2.5.4.  MARKET  SURVEY  OR  INVESTIGATION  NAME.  Use  this 
paragraph  to  describe  a  market  survey  or  investigation  which 
will  provide  data  for  the  evaluation.  Describe  the  details  of 
this  survey  or  investigation  from  the  written  plan  for  the 
effort.  Repeat  this  paragraph  as  necessary  for  multiple 
surveys  and  investigations.  This  paragraph  varies  in  scope 
and  complexity  depending  on  the  size  and  sophistication  of  the 
anticipated  survey  or  investigation.  It  provides  the  basis 
for  understanding  the  value  and  context  of  the  data  derived 
from  this  source. 

2.5.5.  OTHER  STUDY  OR  ANALYSIS.  Use  this  paragraph  to 
describe  a  study  or  other  analytic  effort  to  be  used  to  derive 
data  for  the  evaluation.  Describe  the  details  of  this  study 
or  analysis  the  written  plan  and/or  report  for  the  effort. 
Repeat  this  paragraph  as  necessary  for  multiple  studies  and 
analyses.  This  paragraph  varies  in  scope  and  complexity 
depending  on  the  size  and  sophistication  of  the  anticipated 
study  or  analysis.  It  provides  the  basis  for  understanding 
the  value  and  context  of  the  data  derived  from  this  source. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
Documentation  of  Test  and  Evaluation  Plans,  Chapter  2  (cont) 
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DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  2 — EVALUATION  PLAN  (cont) 

2.6.  DATA  BASE  STRUCTURE.  The  evaluator  describes  a  data 
base  file  structure  that  will  enable  him  to  analyze  test  data 
in  a  thorough  and  timely  manner  and  to  integrate  it  with  data 
from  other  sources. 

2.6.1.  IDENTIFICATION  OF  REQUIRED  FILES.  The  evaluator 
specifies  the  files  required  and  labels  each  with  a  short 
description  of  the  type  of  data  to  be  stored  in  each  file.  He 
also  indicates  for  each  file  whether  it  is  to  be  automated. - 

2.6.2.  DESCRIPTION  OF  FILE  RELATIONSHIPS.  Describe  the 
architecture  used  to  organize  the  data  base.  The  architecture 
typically  consists  of  either  a  network  or  relational  design 
for  data  storage. 

2.6.3.  DATA  ELEMENT  DEFINITIONS.  For  each  file,  the 
evaluator  lists  known  data  elements  and  provides  sufficient 
definitions  to  preclude  ambiguity  and  misunderstanding. 

2.7.  EVALUATION  LIMITATIONS.  This  paragraph  states  known 
limitations  of  the  evaluation  and  those  data  sources  (to 
include  the  operational  test)  which  support  the  evaluation. 
These  limitations  are  to  be  presented  and  discussed  in  terms 
of  the  impact  of  the  limitations.  This  will  include  an 
explanation  of  why  the  limitations  exist,  which  COIC  are 
impacted,  and  what  can  be  done  to  minimize  their  impact. 


Figure  3-24.  Detailed  guidance  for  the  development  and 
Documentation  of  Test  and  Evaluation  Plans,  Chapter  2  (cont) 
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Issue  1.  How  well  do  the  proposed  scout  platoons  perform 
compared  to  the  standard  scout  platoons? 

Question  1.1 

Question  1.2 

Question  1 . 3 

How  effective  are 
the  two  platoons 
in  reconnaissance 
operations? 

How  effective  are 
the  two  platoons 
in  security 
operations? 

How  effective  are 
the  two  platoons 
in  economy-of- 
force  operations? 

Figure  3-25. 

Example  of  a  subdivision 
by  situations 

of  a  test  issue 
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4.2  What  are  the  logistics  implications  of  weapons  system  A? 


4.2.1 

What  are  the  supply 
requirements  of 
weapons  system  A? 


Figure  3-26. 


4.2.2 


4.2.3 


What  are  the  mainte¬ 
nance  requirements  of 
weapons  system  A? 


What  are  the 
transportation 
requirements  of 
weapons  system  A? 


Example  of  a  division  of  a  subquestion 
by  component  attributes 


3.7.5  Is  weapon  system  A  maintainable? 


3. 7. 5.1 

What  was  the  aver, 
no.  of  rounds  fired 
between  failures? 


3. 7. 5. 2 

What  was  the  aver, 
working  time  req'd 
to  repair  failures? 


3. 7. 5. 3 

What  was  the  ratio 
of  maintenance  hrs 
to  rounds  fired? 


Figure  3-27.  Example  of  division  of  a  subquestion  into  measures 


3. 7. 5.1  What  was  the  average  number  of  rounds  fired  between 
failures? 

3. 7. 5. 1.1  How  many  rounds  were  fired  during  each  firing 
attempt  before  a  failure  occurred? 

DE  3. 7. 5. 1.1.1  Did  the  weapon  successfully  fire? 

Figure  3-28.  Example  of  subdivision  to  DE  level,  example  1 
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3. 7. 5. 2  What  was  the  average  working  time  required  to  repair 
failures? 


3. 7. 5. 2.1  What  was  the  working  time  required  to  repair  each 
failure? 

DE  3. 7. 5. 2. 1.1  What  time  did  the  armorer  begin  each  work 
period? 

DE  3. 7. 5. 2. 1.2  What  time  did  the  armorer  end  each  work 
period? 

What  was  the  numerical  designation  of  the 
on  during  this  period? 


Figure  3-29.  Example  of  subdivision  to  DE  level,  example  2 


DE  3. 7. 5. 2. 1.3 
failure  worked 


3. 7. 5. 3  What  was  the  ratio  of  maintenance  hours  to  rounds 
fired? 

3. 7. 5. 3.1  What  was  the  total  number  of  hours  devoted  to 
maintenance? 

DE  3. 7. 5. 3. 1.1  What  time  did  the  gunner  or  armorer  begin  each 
maintenance-  period? 

DE  3. 7. 5. 3. 1.2  What  time  did  the  gunner  or  armorer  end  each 
maintenance  period? 

3. 7. 5. 3. 2  What  was  the  total  number  of  rounds  fired?  ^ 

DE  3. 7. 5. 3. 2.1  Did  the  weapon  fire  successfully  each  time?" 


Figure  3-30.  Example  of  subdivision  to  DE  level,  example  3 


1.3. 7. 4  Were  organization  communications  personnel  able  to 
erect  the  antenna  without  difficulty? 

1.3. 7. 4.1*  What  was  the  average  time  required  to  erect  the 
antenna? 

1 . 3 . 7 . 4 . 2*  What  percentage  of  the  test  personnel  responded 
that  erecting  the  antenna  was  ’'difficult”? 

1 . 3 . 7 . 4 . 3**  What  were  the  comments  of  test  personnel  on  the 
difficulty  involved  in  erecting  the  antenna? 

•MOP. 

‘l^on-MOP . 


Figure  3-31. 


Example  of  subquestion  not  leading  to  MOP 


Issue  3  What 
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Question  3*1  What.. 

3.1.1  - ? 

3. 1.1.1  - ? 

DE  3. 1.1. 1.1  - ’ 

DE  3. 1.1. 1.2  ....? 

3. 1.1. 2  - ? 

DE  3. 1.1. 2.1  ....? 

DE  3. 1.1. 2. 2  ....? 


3. 1.2.1  ....? 

3. 1.2. 1.1  ...."? 

3. 1.2. 1.1.1  ....? 

3. 1.2. 1.1. 1.1  - ’ 

DE  3. 1.2. 1.1. 1.1.1  ....? 

DE  3. 1.2. 1.1. 1.1. 2  - ? 


Figure  3-32.  Example  of  PA  format  in  text 
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Figure  3-33.  Example  of  dendritic  tree  format  for  a  PA. 


OPERATIONAL  TEST 


Purpose;  To  provide  data  on  the  operational 
effectiveness  of  weapon  system  A.  Results  will  be 
used  as  input  to  the  in-process  review  as  a  basis 
for  a  decision  on  continued  development. 

FORCE  DEVELOPMENT  TEST  (CONCEPT) 

Purpose:  To  evaluate  the  effectiveness  of  the 
airborne  rifle  company  organized  under  proposed  TOE 
7-947T.  Test  results  will  be  used  by  TRADOC  as  a 
basis  for  possible  TOE  changes. 

FORCE  DEVELOPMENT  TEST  (MATERIEL) 

Purpose:  To  evaluate  the  military  potential  of  the 

helicopter  storage  system.  Results  will  be  used  by 
AMC  to  assess  the  utility  of  enclosed  storage 
systems  for  helicopters  and,  if  appropriate,  as 
input  for  a  required  operational  capability  (ROC) 
document . 

CONCEPT  EVALUATION  PROGRAM 

Purpose:  To  evaluate  the  compatibility  for  the  Generic 

Aviation  Nitrogen  Generator  (GANG)  with  the  AH-1,  AH-64, 
CH-47,  and  UH-60  aircraft  and  nitrogen  bottles.  The 
results  will  be  used  to  determine  user  acceptance  and  to 
identify  future  test  requirements. 

CUSTOMER  TEST 

To  provide  trajectory  data  on  parachute  loads  requiring 
four  G-llB  parachutes  equipped  with  two  2-second  reefing 
line  cutters  and  100-foot  center  lines.  Data  will  be 
provided  to  U.S.  Army  Natick  Research,  Developirent ,  and 
Engineering  Center  (NRDEC)  to  assist  in  determ  ing  if 
airdrop  configuration  changes  are  feasible. 


Figure  3-34.  Examples  of  purpose  statements 
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TABLE  3-1. 

CLASSIFICATION  OF  TEST  VARIABLES 

Independent 

Dependent 

Extraneous 

Range  of  engagement 

Percentage  of  hits 

MOS  of  gunners 

Target  behavior 

Miss  distance 

Level  of  training 

Light  conditions 

Number  of  weapon 

Weather 

Type  weapon 

failures 

Wind 

Weapon  stabilization 

User  acceptance 

Indiv  marksmanship 
capability 
Leadership 

Figure  3-35.  Example  of  classification  of  test  variables 


Figure  3-36.  Example  of  treatment  of  independent  variables 


Factors 

Control 

Conditions 

Range  of  engagement 

Systematically  varied 

100-500,  501-900, 
901-1,300  meters 

Light  conditions 

Systematically  varied 

Day,  night 

Target  movement 

Systematically  varied 

Moving,  stationary 

Threat  arrays 

Systematically  varied 

lAW  threat  support 
package 

NBC 

Systematically  varied 

No  MOPP,  MOPP  2, 

MOPP  4 

Terrain  (phase  I) 

Systematically  varied 

Flat,  rolling 

Terrain  (phase  II) 

Tactically  varied 

Rugged ,  swamp 

Enemy  action 

Systematically  varied 

Attack,  defend 

Battlefield  obs¬ 
curation 

Systematically  varied 

No  smoke,  smoke 

EW  environment 

Systematically  varied 

lAW  threat  support 
package 

Personnel 

Held  constant 

5th-95th  percentile 

Organization 

Held  constant 

Battery  level 

Doctrine/tactics 

Held  constant 

lAW  D&O  support 
package  or  lAW 
TRADOC  support 
package 

Logistics  support 

Held  constant 

ORG,  DS 

Communications 

status 

Tactically  varied 

Radio-voice, 

radio-digital 

Enemy  target 

Tactically  varied 

Troops,  vehicle 
bunker 

Weather 

Uncontrolled 

Rain,  dry,  snow 

System  operating 
status 

Uncontrolled 

Fully  operational, 
degraded  mobility, 
degraded  firepower, 
non-operational 

Figure  3-37.  Sample  list  of  typical  factors  and  conditions 


Phase 

Test  phase  matrix 
conditions 

Events 

1 .  Maneuver 

Tactical,  RTCA 

Attack,  defense 
meeting  engagement 

2.  Live  fire 

Day,  night 

Moving,  stationary 
targets 

3 .  Transport¬ 

Tactical,  strategies 

Aircraft,  rail. 

ability 

wheel  loading 

Figure  3-38.  Sample  test  phase  matrix 


Tank 

type 

200  meter 

Moving 

trial 

Rds  num 

range 

Still 

trial 

Rds  num 

700  meter  range 

Moving  Still 

trial  trial 

Rds  num  Rds  num 

Day 

M60 

20  8 

20 

10 

20 

5 

20 

3 

Ml 

20  4 

20 

13 

20 

16 

20 

15 

Night 

M60 

15  1 

15 

11 

15 

12 

15 

7 

Ml 

15  9 

15 

6 

15 

14 

15 

2 

NOTE: 

Trial  numbers 

indicate  the  randomized 

order  in 

which 

trials 

will  be  performed  so 

that  the 

order 

of 

events  will  be 

mixed 

for  different  crews. 

Figure  3- 

■39. 

Sample  of 

event 

matrix 

Event 

number 

Date /time 

Location 

Descriotion 

38 

250400 

GM  901456 

2d  Platoon,  3/7  Inf  departs 
assembly  area  on  mission  to 
recon  area  WHITE. 

39 

250615 

GM  345564 

2d  Platoon,  3/7  Inf  receives 
change  in  mission  from 

Company  CO  by  radio.  New 
mission  is  to  conduct  recon 
of  area  BLUE. 

40 

250830 

GM  347895 

2d  Platoon,  3/7  Inf,  enters 
meeting  engagement  with  2d 

Co,  4th  Fusilier  Tank  Bn  at 
checkpoint  7. 

Figure  3-40.  Suggested  test  scenario  format 
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45.  Problem:  Friendly  unit  plans  to  attack  positions 
considered  occupied  in  the  scenario  but  not  actually  manned  by 
OPFOR. 

Possible  solutions,  in  order  of  desirability: 

(a)  Move  an  OPFOR  unit  into  the  area. 

(b)  Pass  intelligence  of  barrier  to  induce  change  of 

plan. 

(c)  Allow  attack  of  unmanned  position. 

(d)  Order  friendly  unit  to  change  plan  (last  resort) . 

Figure  3-41.  Example  scenario  revision  guidelines 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  3 — TEST  DESIGN 

CHAPTER  3.  TEST  DESIGN.  This  chapter  is  prepared  by  the 
tester  except  when  the  TEP  is  used  to  plan  evaluation  for  a 
milestone  that  will  not  be  preceded  by  a  dedicated  phase  of 
user  testing.  In  those  cases,  the  evaluator  will  provide  a 
brief  statement  to  that  effect  and  the  remainder  of  chapter  3 
is  eliminated.  Use  appendixes  only  as  necessary  where  the 
material  for  a  specific  paragraph  is  too  voluminous  to  be 
included  in  this  paragraph  without  a  break  in  the  logical  flow 
of  information.  Whenever  an  appendix  is  used,  provide 
summarized  information  within  the  basic  paragraph. 

3.1.  TEST  DESCRIPTION.  Describe  the  test  to  be  conducted  to 
address  the  operational  issues,  OTMOP,  and  data  requirements 
specified  in  paragraph  3.2. 

3.1.1.  TEST  PURPOSE.  This  paragraph  contains  the  purpose  of 
the  test  and  where  the  results  will  be  used.  This  will  be  the 
same  as  the  test  purpose  contained  in  the  OTP. 

3.1.2.  TEST  AUTHORITY .  The  authority  to  conduct  the  test 
will  be  outlined  here  and  will  consist  of  a  reference  to  the 
DA  memorandum  approving  the  current  FYTP  and  the  number  of  the 
approved  OTP. 

3.1.3.  TEST  OVERVIEW.  This  portion  should  provide  an 
overview  of  the  test  design.  The  purpose  of  the  test  design 
is  to  execute  the  concepts  described  by  the  lOE  in  chapter  2 
of  the  TEP. 

3. 1.3.1.  TEST  PHASES.  Based  on  the  test  approach  in  chapter 
2  of  the  TEP,  the  tester  describes  the  tactical  context  and 
the  associated  scenario,  environment,  threat,  tactics,  and 
doctrine  to  be  used  in  each  test  phase. 

3.3  .2.  FACTORS  AND  CONDITIONS.  The  factors  and  conditions 

de  T-d  in  chapter  2  of  the  TEP  are  reviewed  and  refined.  As 
nec  sary,  the  tester,  in  conjunction  with  the  operational 
eva-uitor,  refines  the  factors  and  conditions  by  adding 
factors.  For  a  given  factor,  the  tester  refines  the  factors 
by  changing  the  number  of  conditions,  redefining  the 
conditions  or  changing  the  type  of  control  (systematically 
varied,  tactically  varied,  uncontrolled,  or  held  constant) . 


Figure  3-42.  Detailed  guidance  for  the  development  and 
documentation  of  Test  and  Evaluation  Plans,  Chapter  3 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  3 — TEST  DESIGN  (cont) 

3.2.  CONDUCT  OF  TEST.  Describe  the  conduct  of  the  test  in 
terms  of  its  tactical  context,  events,  control,  phasing, 
scheduling,  and  methodology. 

3.2.1.  TACTICAL  CONTEXT.  Based  on  the  test  concept  in 
chapter  2  of  the  TEP,  describe  the  tactical  context  and 
associated  scenarios,  environment,  threat,  tactics,  and 
doctrine  to  be  used  in  each  test  phase. 

3. 2. 1.1.  SCENARIO(S).  The  scenario  description  consists  of 
the  geographic  portrayal  of  the  area,  forces,  and  events  of  a 
hypothetical  armed  conflict. 

3.2. 1.2.  TEST  ENVIRONMENT.  The  use  of  test  site  terrain  is 
described  for  each  phase,  scenario,  mission,  or  trial. 

3. 2. 1.3.  THREAT  FOR  TEST.  Describe  how  the  threat  systems 
and  tactics  will  be  played  in  each  phase.  An  approved  threat 
will  be  used  in  accordance  with  AR  381-11.  The  threat  will  be 
described  in  sufficient  detail  to  permit  realistic  testing. 

3. 2. 1.4.  TACTICS  AND  DOCTRINE.  This  paragraph  describes  the 
friendly  tactics  and  doctrine  to  be  played  in  each  phase.  The 
phases  are  designed  within  the  tactical  and  doctrinal 
framework  of  the  D&OTSP. 

3. 2. 1.5.  TEST  tJNIT  ORGANIZATION.  Data  on  test  player  forces 
that  will  operate  the  system  and  portray  the  supporting, 
supported,  and  adjacent  forces  in  play.  Identify  type  of  test 
unit  or  organization  for  the  test.  Discuss  how  the  unit  is 
organized  any  significant  requirements  of  the  unit.  Include 
additional  information  such  as  TOE  designation,  as  required. 

3.2.2.  TEST  EVENTS.  Include  discussions  of  the  organization 
and  overall  layout  of  the  test  to  include  sequence  of  phases. 
Flow  diagrams,  time  lines,  and  matrixes  are  used  as 
appropriate  to  introduce  the  events  described. 

3. 2. 2.1.  MISSION  AND  VIGNETTE /BATTLE  DESCRIPTION.  A 
description  of  each  distinct  mission,  vignette,  or  battle 
which  is  a  part  of  the  overall  test. 


Figure  3-42.  Detailed  guidance  for  the  development  and 
documentation  of  Test  and  Evaluation  Plans,  Chapter  3  (cont) 


( 


'^'o:  ^at  Mawbv 


rom:  Kenne+h  L.  Wilson 


11-30-55  4:07Dm 


D.  4  of  7 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  3— TEST  DESIGN  (cont) 

3. 2. 2. 2.  SITE  SPECIFIC  VIGNETTES/BATTLES.  A  list  of  roission, 
vignettes,  or  battles  by  test  site,  range,  or  route.  This  may 

®  figure  or  a  table  called  out  by  this  paragraph.  Terrain 
maps  may  be  used  as  figures  in  the  test  to  depict  the  various 
vignettes/battles . 

3. 2. 2. 3.  VIGNETTE /BATTLE  LIST.  An  overall  list  of  missions, 
vignettes,  ahd  battles.  This  may  be  a  figure  or  a  table  which 
is  called  out  by  this  paragraph. 

3. 2. 2. 4.  EVENT  MATRIX (ES) .  Matrices  necessary  to  organize 
the  test  events.  These  matrixes  may  appear  in  the  test  as 
figures  or  tables  called  out  in  this  paragraph. 

3. 2. 2. 5.  EVENT  SEQUENCING.  A  time-phased  listing  of  events. 
This  may  be  a  figure  or  a  table  which  is  called  out  by  this 
paragraph. 

3.2.3.  CONTROL  PROCEDURES.  Include  descriptions  of  the 
control  structure  and  procedures  to  be  used  to  ensure  that 
required  test  events  occur  in  situations  that  realistically 
depict  the  tactical  context  of  the  test  in  accordance  with  the 
OMS/MP.  The  paragraph  ties  together  control  requirements  for 
each  OTMOP  in  paragraph  3.2.  The  control  procedures  are 
expanded  into  a  Control  Concept  in  Appendix  J. 

3.2.4.  SCHEDULE  OF  EVENTS.  Overall  test  schedule.  This 
information  may  appear  in  a  figure  or  table  called  out  in  this 
paragraph . 

3.2.5.  OVERALL  METHODOLOGY.  Include  descriptions  of 
execution  procedures,  control  procedures,  data  collection 
procedures,  and  any  other  information  necessary  for  an 
understanding  of  the  matrix  that  is  applicable  to  all  issues. 

3.2.6.  TEST  LIMITATIONS.  All  known  limitations  to  the 
adequacy  of  the  test  (such  as  duration,  number  of  systems, 
availability  of  interoperable  systems,  test  unit  availability, 
and  ITTS  and  their  impact  on  the  OTMOPs  will  be  described. 


Figure  3-42.  Detailed  guidance  for  the  development  and 
documentation  of  Test  and  Evaluation  Plans,  Chapter  3  (cont) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
CHAPTER  3— TEST  DESIGN  (cont) 

3.3.  TEST  DETAILS.  This  paragraph  is  a  thorough  description 
of  how  the  tester  intends  to  provide  the  OTMOP.  Arrangement 
should  be  such  that  major  paragraphs  are  presented  for  each 
issue.  This  organization  is,  therefore,  directly  aligned  with 

I  issues  in  TEP  chapter  2.  Each  operational  issue  in  paragraph 
2.2  that  contains  one  or  more  OTMOP  is  an  issue  to  be 
addressed  in  OT.  Issue  numbers  should  carry  forward,  so  that 
there  may  be  gaps  in  the  issue  numbering  which  reflects 
paragraph  2.2  issues  not  addressed  in  the  OT.  For  abbreviated 
evaluations  and  for  testing  of  CBTDEV  and  TNGDEV  products,  all 
of  the  MOP  developed  in  chapter  2  will  be  OTMOP. 

3.3.1.  ISSUE  1.  This  is  a  restatement  of  issue  1  as  listed 
in  chapter  2  of  the  TEP. 

3. 3. 1.1.  OTMOP  1“1*  This  is  the  first  OTMOP  associated  with 
issue  1.  The  criteria  associated  with  this  issue  are  not 
listed.  If  the  events,  control  mechanisms,  collection 
methods,  reduction  methods,  and  analysis  methods  are  closely 
aligned  for  all  OTMOP  associated  with  this  test  issue, 
paragraphs  that  follow  only  need  to  be  stated  once.  If  not 
aligned,  the  following  paragraphs  must  be  developed  for  each 
OTMOP . 

a.  METHODOLOGY.  This  paragraph  describes  the  test  events 
that  must  be  executed  to  generate  the  data  required,  the 
operational  conditions  under  which  the  events  must  occur,  the 
techniques  for  collecting  the  data,  and  the  control  procedures 
that  will  be  used  to  ensure  that  test  events  occur  at  the 
proper  time  and  place  and  under  the  conditions  specified. 

Where  appropriate,  reference  can  be  made  to  information 
previously  presented  in  paragraph  3.2.5  of  the  TEP. 

(1)  TEST  EVENTS.  Required  events  should  be  listed  or 
referenced  to  the  appropriate  subparagraph  of  paragraph  3.2.2 
of  the  TEP. 

(2)  CONTROL  CONCEPTS.  Describe  the  specific  control 
procedures  and  rules  of  engagement  that  will  be  employed  to 
ensure  that  required  test  events  occur  in  situations  that 
realistically  depict  the  tactical  context  of  the  test.  Specify 
the  level  of  operational  realism  required  in  the  test  (full 
tactical  simulation,  limited  tactical  simulation,  or  no 
tactical  simulation)  and  the  rationale  for  the  choice. 
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b.  DATA  REQUIREMENTS.  Data  requirements  applicable  to  each 
OTMOP  should  be  extracted  from  Appendix  G,  Pattern  of 
Analysis,  and  summarized  here.  This  section  should  not  be 
merely  a  repeat  of  the  information  in  appendix  G.  If  no  new 
information  is  to  be  presented,  reference  can  be  made  to  the 
appropriate  section  of  appendix  G  for  data  requirements. 
Information  concerning  data  accuracies  and  sample  sizes  not 
presented  in  paragraph  3.1  or  in  chapter  2  of  the  TEP  should 
also  be  presented  here. 

c.  DATA  COLLECTION.  Describe  the  procedures  for  the 
collection  and  tabulation  of  test  data  and  the  specific 
organization  and  procedures  to  be  used  to  collect  the  reguired 
data. 

d.  DATA  REDUCTION.  Describe  the  procedures  for  the 
processing  of  test  data.  Each  method  to  be  employed  to 
process  data  required  to  answer  the  OTMOP  should  be  addressed. 

e.  DATA  MIALYSIS.  Describe  the  logic  for  formulating  and 
computing  findings  and  formulating  any  assessments  which  might 
be  required. 

3 . 3 . 1 . 2 .  OTMOP  1-2 . 

through 

3.3.1. x.  OTMOP  1-x. 

3.3.2.  ISSUE  2. 
through 

3.3. n.  ISSUE  n. 

3.4.  TEST  DATA  MANAGEMENT.  Provide  the  concepts  for  dat^ 
management  and  the  requirements  for  the  tester  data  base. 

3.4.1.  DATA  COLLECTION  CONCEPT.  Describe  the  test  element 
organization  and  responsibilities  for  data  collection  and 
instrumentation  operation  and  maintenance.  Delineate  the 
requirements  for  data  collector  training  on  the  test  item  and 
use  of  data  collection  equipment. 
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3.4.2.  DATA  REDUCTION  CONCEPT.  Describe  the  process  in  which 
recorded  test  data  is  organized,  reduced,  verified,  managed, 
controlled,  and  stored.  Organize  the  data  in  terms  of  the 
collection  source  (RAM  collection  form,  cockpit  digital 
recorder,  radar  tapes,  etc.).  Diagram  the  process  through 
which  each  set  of  collected  data  is  to  pass  before  reaching 
the  storage  medium  which  supports  the  test  directorate  and  the 
evaluator.  This  data  flow  diagram  identifies  where  data  is 
combined  with  other  data,  where  it  is  processed,  scored, 
reorganized,  validated,  or  otherwise  manipulated. 

3.4.3.  QUALITY  CONTROL  CONCEPT.  Outline  the  process  for 
independently  validating  test  data.  Define  the  checks  and 
procedures  which  are  to  be  used  to  preclude  or  detect  and 
correct  errors  made  in  data  collection,  data  entry,  or  data 
reduction.  It  also  identifies  emerging  data  summaries  required 
to  identify  potential  inconsistencies. 

3.4.4.  DATA  REDUCTION  TIMELINES.  Provide  a  timeline  flow  of 
the  various  data  collection  processes. 

3.4.5.  DATA  BASE  DESIGN.  When  test  data  are  extensive  enough 
to  require  storage  in  an  automated  data  base,  the  structure 
and  content  of  the  data  base  is  described  in  this  section. 

3. 4. 5.1.  DATA  ELEMENT  DICTIONARY.  The  data  in  each  file  or 
record  are  to  be  listed  and  augmented  by  any  necessary 
definitions . 

3. 4. 5. 2.  DATA  BASE  STRUCTURE.  Provide  data  base  structure 
diagrams  which  expand  on  data  base  descriptions  in  chapter  2. 

3. 4. 5. 3.  DATA  COLLECTION  FORMS  MATRIX.  Provide  a  crosswalk 
of  data  elements  and  the  data  collection  forms  oh  which  these 
data  elements  will  be  collected. 

3.5.  TEST  PERSONNEL,  EQUIPMENT,  AND  MATERIALS. 

3.5.1.  TEST  DIRECTORATE  ORGANIZATION.  Describe  the  makeup 
and  general  organization  of  the  test  directorate  required  to 
execute  the  test  as  described  in  the  test. 


Figure  3-42.  Detailed  guidance  for  the  development  and 
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3.5.2.  TRAINING.  This  paragraph  describes  besb  braining 
requireinen'ts .  Training  re^iremenbs  (acbual  braining  and 
necessary  resources)  consisb  of  bwo  major  elemenbs:  braining 
bhe  besb  players  and  braining  bhe  besb  organizabion  personnel. 

3. 5. 2.1.  TEST  PLAYER  PERSONNEL.  A  summary  of  braining  bo  be 
given  bo  personnel  or  unibs  in  preparabion  for  cerbif icabion 
(OTRS)  and  besbing  should  be  presenbed. 

3. 5. 2. 2.  TEST  DIRECTORATE  PERSONNEL.  Training  bo  be 
presenbed  bo  conbrollers,  daba  collecbors  and  reducers,  and 
obher  besb  personnel  should  also  be  summarized. 

3.5.3.  EQUIPMENT  AND  MATERIALS. 

3.5.3. 1.  INSTRUMENTATION.  Describe  bhe  insbrumenbabion 
required  bo  supporb  bhe  besb. 

3. 5. 3. 2.  TARGETS  AND  THREAT  SIMULATORS.  Describe  bhe  bargebs 
and  bhreab  siroulabors  required  bo  supporb  bhe  besb. 

3.5.3. 3.  ADP  EQUIPMENT.  Describe  bhe  ADP  equipmenb  and 
archibecbure  required  bo  supporb  bhe  besb. 

3. 5. 3. 4.  AUDIOVISUAL  EQUIPMENT.  Describe  bhe  audiovisual 
equipmenb  required  bo  supporb  bhe  besb. 

3. 5. 3. 5.  TEST  SUPPORT  EQUIPMENT.  Describe  bhe  all  equipmenb, 
maberiel,  and  services  required  bo  supporb  bhe  besb  nob 
obherwise  enumerabed  above. 

3. 5. 3. 6.  TEST  SUPPORT  FACILITIES.  Describe  bhe  base  and 
range  facilibies  (B&RF)  required  bo  supporb  bhe  besb. 


3.6.  ENVIRONMENTAL  AND  ENERGY  IMPAC'' 
environmenbal  and  energy  impacbs  fro! 


Summarize  bhe 
pendixes  P  and  Q. 


Figure  3-42.  Debailed  guidance  for  bhe  developmenb  and 
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TEST  CHANGE  PROPOSAL  FOR  FULL-EVALUATE  TEP 

Date : 

Tested 

System 

Medium  Tactical  Wheel  (MTW) ,  lOTE  91-OT  XXXX- 
0001 

Proposed  By 

Major  Hotspur  J.  Treadhead,  CSTE-EXX, 

Operational  Evaluation  Command  (Operational 
Evaluator) 

Proposed 

Change 

Terminate  MTW  night  operations.  Convert 
remaining  night-test  periods  to  day-time  test 
periods.  Total  test  length  will  remain 
unchanged. 

Rationale 

The  change  is  desired  in  order  to  provide  an 
opportunity  to  increase  test  mileage  and 
facilitate  meeting  lOTE  RAM  mileage  goals. 

Resource 

Impacts 

A:  Time.  Reduce  night  operations  from  the 

originally  planned  27  days  to  21  days  of  test 
time. 

B:  Cost.  Unknown  at  this  time. 

C:  Personnel.  No  impact. 

Other 

Impacts 

None. 

Coord 

Verbal  coordination  has  been  performed  with 
headquarters,  DEC;  Office  of  the  DUSA(OR) ;  and 
Office  of  the  DOTE.  Have  obtained  concurrence. 

APPROVAL: 

TEXCOM  OEC 

IZ2Y.  A.  TESTER  LORENZO  P.  OVERVUE 

BG,  USA  Col,  AG 

Commanding  Commanding 

RELEASED: 

United  States  Army 

Operational  Test  and  Evaluation  Command 

WILLIE  J.  TOPDOG 

Major  General,  USA 

Commanding 

ATTACHMENTS 

:  Revised  page  of  TEP 

Figure  3-43.  TCP  format  for  full-evaluate  level  TEP 


TEST  CHANGE 

PROPOSAL  FOR  OTHER  THAN  FULL-EVALUATE  LEVEL  TEP 

Date: 

Tested 

System 

Medium  Tactical  Wheel  (MTW) ,  lOTE  91-OT-XXXX- 
0001 

Proposed  By 

LTC  Hotspur  J.  Treadhead,  CSTE-EXX,  Operational 
Evaluation  Command  (Operational  Evaluator) 

Proposed 

Terminate  MTW  night  operations.  Convert  Change 
remaining  night-test  periods  to  day-time  test 
periods.  Total  test  length  will  remain 
unchanged. 

Rationale 

The  change  is  desired  in  order  to  provide  an 
opportunity  to  increase  test  mileage  and 
facilitate  meeting  lOTE  RAM  mileage  goals. 

Resource 

Impacts 

A:  Time.  Reduce  night  operations  from  the 
originally  planned  27  days  to  21  days  of  test 
time. 

B:  Cost.  Unknown  at  this  time. 

C:  Personnel.  No  impact. 

Other 

Impacts 

None. 

Coord 

Verbal  coordination  has  been  performed  with 
headquarters,  OEC;  Office  of  the  DUSA(OR);  and 
Office  of  the  DOTE.  Concurrence  has  been 
obtained. 

APPROVAL: 

IZZY  A.  TESTER 

BG,  USA 

Commanding 

ATTACHMENTS : 

Revised  page  of  TEP 

Figure  3-44.  TCP  format  for  other  than  full-evaluate  level  TEP 
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COORDINATION: 

Test  and  Experimentation  Command 

Opnl  Tester 

Concur / nonconcur 

(Test  Officer/MAJ) 

INIT:  DATE 

Opnl  Analyst 
INIT:  DATE_ 

Concur / nonconcur 

(Analyst/CPT) 

Operational  Evaluation  Command 

Opnl  Evaluator 
INIT:  DATE_ 

Concur / nonconcur 

(Evaluator/LTC) 

Eval  Analyst 
INIT:  DATE_ 

Concur / nonconcur 

(Analyst/GM14) 

Figure  3-45.  Tester/evaluator  coordination  block  for 

TCP  submission 


( 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 

APPENDIXES 

APPENDIXES: ^  Some  appendixes  are  provided  by  the  evaluator. 
Some  appendixes  are  provided  by  the  tester.  Responsibility 
for  each  appendix  is  annotated  below.  Any  appendix  shown  as 
optional  is  not  required  if  the  information  can  be  adequately 
addressed  in  the  body  of  the  TEP.  If  an  appendix  is  not  used, 
show  that  by  a  place  holder.  Do  not  redesignate  appendixes. 

APPENDIX  A  “  SUPPORTING  DOCUMENTATION.  This  required  appendix 
is  developed  by  the  evaluator  and  provides  a  listing  of 
P®rtinent  system  documentation  to  include,  as  a  minimum,  the 
MNS,  ORD,  TEMP,  OTP,  TSP,  and  the  TEA  and  EIS  (if  applicable) . 
Each  document  is  listed  by  title,  responsible  activity,  status 

approved,  draft,  under  revision),  and  date  of  approval 
or  draft. 

APPENDIX  B  “  BACKGROUND.  This  optional  appendix  is  prepared 
the  evaluator  and  contains  details  of  the  background  of  the 
system  and  T&E  of  the  system.  If  the  details  to  be  placed  in 
paragraph  1.3  are  so  voluminous  that  TEP  continuity  is 
lost,  they  are  summarized  in  the  TEP  with  details  in  appendix 
B.  Appendix  B  is  optional  if  details  are  adequately  covered 
in  the  TEP. 

APPENDIX  C  —  SYSTEM  DESCRIPTION.  This  optional  appendix  is 
prepared  by  the  evaluator  and  includes  a  description  of  the 
system  (or  CBTDEV/TNGDEV  product) .  If  a  large  amount  of 
detail  is  required  to  adequately  understand  the  system,  a 
lengthy  system  description  is  included  as  Appendix  C. 

Appendix  C  is  optional  if  the  details  are  be  adequately 
covered  in  the  TEP. 

APPENDIX  D  -  PROJECTED  THREAT.  This  optional  appendix  is 
prepared  by  the  evaluator  and  defines  the  approved  threat  in 
the  post-IOC  timeframe  of  the  tested  system.  If  a  large 
amount  of  detail  is  required  to  adequately  understand  the 

®  lengthy  threat  statement  is  included  as  Appendix  D. 
Appendix  D  is  optional  if  the  details  can  be  adequately 
covered  in  the  TEP. 

APPENDIX  E  —  DAG  CHARTER  AND  SOP.  The  evaluator  includes  a 
copy  of  the  DAG  Charter  and  SOP  if  a  DAG  requirement  is 
specified  in  TEP  chapter  2.  For  abbreviated  evaluations  or 
tests  of  CBTDEV  or  TNGDEV  products,  the  tester  may  use  a  DAG 
and  add  this  appendix. 
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DETAILED  GUIDANCE  EOF  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
APPENDIXES  (continued) 

I  APPENDIX  F  -  OUTLINE  TEST  PLAN  (OTP).  A  copy  of  the  most 
recent  and  approved  OTP  referenced  in  Appendix  A  is  included 
as  Appendix  F  by  the  tester.  Where  appropriate,  a  test  resume 
sheet  (TRS)  may  be  substituted  for  the  OTP. 

APPENDIX  G  -  PATTERN  OF  ANALYSIS.  The  POA  is  the  tester's 
detailed  refinement  of  each  operational  test  issue  and 
associated  criteria  into  OTMOPs  and  data  elements.  It  should 
be  developed  as  the  first  step  in  the  preparation  of  the  test 
design  for  all  tests.  The  final  POA  is  included  here. 

APPENDIX  H  -  TEST  SCENARIO (S).  This  optional  appendix 
contains  details  of  the  test  scenario  not  contained  in  the 
main  body  of  the  document. 

APPENDIX  I  -  TEST  THREAT.  This  optional  appendix  contains 
threat  portrayal  details  not  contained  in  the  main  body  of  the 
TEP. 

APPENDIX  J  -  CONTROL  CONCEPT.  Include  descriptions  of  the 
control  structure  and  procedures  that  will  be  used  to  ensure 
that  reguired  test  events  occur  in  situations  that 
realistically  depict  the  tactical  context  of  the  test  lAW  the 
OMS/MP.  The  control  concept  is  a  preview  of  the  Control  Plan 
in  the  DTP. 

APPENDIX  K  -  DATA  MANAGEMENT  CONCEPT.  Outlines  test 
j  organization  and  responsibilities  for  data  management.  The 
data  management  concept  is  a  preview  of  the  Data  Management 
Plan  of  the  DTP.  The  following  information  is  to  be  included. 

APPENDIX  L  -  INSTRUMENTATION,  TARGETS,  AND  THREAT  SIMULATORS 
(ITTS)  SUPPORT  CONCEPT.  This  optional  appendix  contains 
details  of  planned  ITTS  support  not  contained  in  the  main  body 
of  the  document.  The  ITTS.  support  concept  is  a  preview  of  the 
ITTS  Support  Plan  in  the  DTP. 

APPENDIX  M  -  AUDIOVISUAL  SUPPORT  CONCEPT.  This  optional 
appendix  contains  details  of  planned  audiovisual  support  not 
contained  in  the  main  body  of  the  document.  The  audiovisual 
support  concept  is  a  preview  of  the  Audiovisual  Support  Plan 
in  the  DTP. 
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APPENDIXES  (continued) 

APPENDIX  N  -  ADP  SUPPORT  CONCEPT.  This  optional  appendix 
contains  details  of  planned  ADP  support  not  contained  in  the 
main  body  of  the  document.  The  ADP  support  concept  is  a 
preview  of  the  ADP  Support  Plan  in  the  DTP. 

APPENDIX  0  -  TRAINING  CONCEPT.  The  training  concept  is 
actually  developed  during  the  evolution  of  the  control  concept 
and  data  management  concept.  This  appendix  contains  details 
of  planned  training  of  both  player  personnel  and  test 
directorate  personnel.  The  training  concept  is  a  preview  of 
the  Training  Plan  in  the  DTP. 

APPENDIX  P  -  TEST  SUPPORT  CONCEPT.  This  optional  appendix 
contains  details  of  planned  test  support  (equipment,  materiel, 
personnel,  and  services)  not  contained  in  the  main  body  of  the 
document.  The  test  support  concept  is  a  preview  of  the  Test 
Support  Plan  in  the  DTP. 

APPENDIX  Q  -  TEST  ENVIRONMENTAL  ASSESSMENT  (TEA) .  The  tester 
includes  a  copy  of  the  approved  TEA  (if  applicable) . 

APPENDIX  R  -  ENVIRONMENTAL  IMPACT  STATEMENT  ^IS) .  The  tester 
includes  a  copy  of  the  approved  EIS  (if  applicable) . 

APPENDIX  S  -  DOTE  APPROVAL  OF  THE  TEST  CONCEPT.  This  appendix 
is  required  for  TEP  for  all  DOTE  oversight  programs.  The 
evaluator  includes  a  copy  of  formal  DOTE  approval  of  the  test 
concept . 

APPENDIX  T  -  TEST  CHANGE  PROPOSALS  (TCP) .  This  required 
appendix  is  added  as  a  placeholder  by  the  tester.  As  TCP  are 
developed  during  the  time  between  TEP  approval  and  end  of  the 
test,  they  are  added  in  chronological  sequence  as  tabs  to  the 
appendix. 

APPENDIX  U  -  GLOSSARY,  ACRONYMS,  AND  ABBREVIA  NS.  This 
appendix  is  prepared  as  required  by  the  evaluate r  and  the 
tester.  Any  unusual  technical  terms  or  frequently  used 
acronyms  and  abbreviations  in  the  body  of  the  TEP  or  in  other 
appendixes  will  be  defined  in  this  appendix  by  the  evaluator 
for  his  portion  of  the  TEP  and  by  the  tester  for  those  terms 
which  he  has  introduced  and  which  were  not  previously  defined 
by  the  evaluator. 
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DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND 
DOCUMENTATION  OF  TEST  AND  EVALUATION  PLANS 
APPENDIXES  (continued) 

APPENDIX  V  -  TEP  COORDINATION  RECORD.  A  listing  of  agencies 
with  which  the  TEP  was  coordinated  and  a  recor(L_of  comment 
resolution. 

APPENDIX  W  -  AUTHORS  AND  SUPPORTING  PERSONNEL.  For  historical 
purposes,  all  individuals  having  input  to  or  knowledge  of  the 
writing  of  the  TEP  should  be  listed  in  this  appendix  by  the 
evaluator  unless  the  listing  of  personnel  on  the  coordination 
sheet  is  sufficient. 

APPENDIX  X  -  DISTRIBUTION.  Distribution  of  the  TEP  by 
organization  and  number  of  copies  is  shown  in  this  appendix  by 
the  evaluator. 


Figure  3-46.  Detailed  guidance  for  the  development  and 
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DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


Chapter  4 

Management  of  Test  and  Evaluation  Resources 


4-1.  Resources  for  operational  testing 

This  chapter  provides  specific  information  for  the  management  of 
resources  to  support  operational,  customer,  and  other  user  tests 
with  emphasis  on  Operational  Test  and  Evaluation  (0T6E) .  , 

includes  details  on  the  processes,  products,  and  responsibilities 
of  Army  organizations  in  the  management  of  test  and  evaluation 
resources.  Resources  are  defined  here  as  the  personnel,  funding, 
materiel,  and  documentation  required  for  effective  test  and 
evaluation. 

4-2.  OT&E  management  processes 

Operational  testing  (OT)  requirements  originate  from  OSD's  Joint 
Testing  Program,  Multiservice  and  Army  Test  and  Evaluation  Master 
Plans  (TEMP),  Army's  Concept  Evaluation  Program  (CEP),  and 
materiel  and  combat  developers  with  special  needs  (customer 
tests) .  The  processes  that  support  OT  planning,  documentation, 
and  execution  include  OSD's  JT&E  and  Feasibility  Study  processes. 
Test  Integration  Working  Groups  (TING) ,  Test  Schedule  and  Review 
Committee  (TSARC)  (see  figure  4-3) ,  Five  Year  Test  Program 
(FYTP),  TRADOC's  CEP  Schedule  and  Review  Committee  (CEPSARC) , 
OPTEC  Operational  Test  Readiness  Reviews  (OTRR) ,  and  OPTEC  Test 
and  Evaluation  Coordination  Offices  (TECO)  and  test  directorates 
located  at  TRADOC  installations.  An  explanation  of  how  these 
processes  work  and  relate  to  each  other  follows. 

4-3.  Joint/Multi-Service  Test  and  Evaluation  (JT&E) 

a.  OSD  Directed  JT&E.  This  type  of  T&E  brings  two  or  more 
services  together  to  evaluate  developmental  or  operational 
concepts,  inter-operability,  methodologies,  models  and 
simulations,  test  beds,  etc.,  as  directed  in  a  formal  charter 
from  the  Deputy  Director,  Defense  Research  and  Engineering  (Test 
and  Evaluation)  (DDDRE(T&E) ) .  The  JT&E  program  is  supported  by 
an  annual  OSD  nomination  process,  a  feasibility  study  process  of 
7-8  months,  and  a  testing  process  of  3  or  more  years. 

(1)  Army  nominations  to  conduct  a  JT&E  may  be  drafted  at 
anytime  by  any  organization.  The  Army's  JT&E  solicitation  and 
nomination  process  is  controlled  and  implemented  by  DA  ODCSOPS 
(DAMO-FDT) .  The  selection  of  suitable  nominations  to  become 
feasibility  studies  and  the  selection  of  completed  feasibility 
studies  to  become  chartered  OSD  directed  JT&E  is  determined 
primarily  by  the  recommendations  of  the  Senior  Advisory  Council 
(SAC),  co-chaired  by  the  DDDRE  T&E,  and  the  Director,  Operational 
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Test  and  Evaluation  (DOT&E) .  The  SAC,  which  includes  Army 
representation  form  DA  ODCSOPS,  reviews  the  nominations  and 
recommends  selected  nominations  for  entry  into  the  feasibility 
study  phase. 

(2)  During  the  next  6-9  months,  the  feasibility  study 
director  supported  by  TDY  AD  Hoc  personnel  identify  the  proposed 
JT&E  details:  issues,  objectives,  milestones,  developmental/ 
operational  requirements,  resources,  funding,  possible  test 
locations,  preliminary  test  design,  and  requirements  for 
simulations,  instrumentation,  joint  exercises,  data  collection, 
and  evaluation.  Upon  completion  of  the  feasibility  study  and 
favorable  review  by  the  SAC,  the  JT&E  candidate  may  be 
recommended  for  charter  as  a  JT&E. 

(3)  JT&E  charters  provide  for  a  "lead  service"  and  one  or 
more  "supporting  services."  The  services  provide  necessary  O&M 
funding,  PCS  personnel,  and  other  TDY  personnel  and  equipment 
support  for  the  JT&E  consistent  with  their  involvement  as  defined 
in  the  approved  feasibility  study. 

(4)  OPTEC  has  overall  Army  management  responsibility  for 
JT&E.  Accordingly,  OPTEC  provides  a  member  to  the  Joint  Test 
Planning  Committee.  This  is  a  working  level  body  which  meets  to 
review  nominations/feasibility  studies,  provide  advice  on  JT&E, 
exchange  service  positions  and  prepare  nominations  and 
recommendations  for  presentation  to  the  SAC.  As  the  resource 
manager  in  support  of  chartered  JT&E,  OPTEC  maintains  manpower 
authorizations  on  the  U.S.  Army  Element  Joint  Test  Activities 
TDA,  requisitions  personnel  to  staff  the  full  time  test 
directorate  positions,  budgets  for  the  Army's  participation  and 
lead  service  costs,  and  coordinates  Army-wide  JT&E  support 
requirements  through  the  TSARC  process.  (See  figure  4-3.) 

Al*  .ugh  OPTEC  is  not  responsible  for  the  detailed  management  of 
th  oint  test  force  (JTF) ,  OPTEC  provides  technical  T&E  advice 
thi  .gh  test  document  reviews,  technical  advisory  groups  (TAGs) , 
General  Officer  Steering  Committees  (GOSC) ,  and  membership  on  the 
OSD  Technical  Advisory  Board  (TAB) . 

b.  Multiservice  Test  and  Evaluation  (NT&E) .  Multiservice 
tests  are  normally  initiated  by  a  Joint  Service  Operational 
Requirement  (JSOR) .  Tests  are  conducted  for  systems  being 
acquired  by  more  than  one  DOD  or  for  systems  which  interface  with 
equipment  of  another  service.  OSD  will  designate  a  lead  service 
which  will  prepare  the  final  report  on  the  system.  However, 
resource  planning  and  support  is  the  same  as  for  any  other  Army 
OT.  Requirements  are  documented,  coordinated,  and  prioritized  in 
the  TSARC  and  FYTP  processes.  As  with  JT&E,  OPTEC  is  the  focal 
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point  for  coordination  of  Army  resources  to  support  multiservice 
test  and  evaluation.  This  includes  budgeting  for  the  testing 
necessary  to  accomplish  assigned  test  objectives  and  for 
participation  of  our  personnel  and  equipment  in  the  entire  test 
program. 

4-4.  Test  and  Evaluation  Master  Plan  (TEMP),  reference:  Part 
Two,  TEMP  Format,  Review  and  Approval  Procedures 
Program  managers  (PM)  are  responsible  for  developing  the  TEMP, 
which  is  a  basic  planning  document  for  all  life  cycle  test  and 
evaluation  activities. 

a.  Key  elements  of  the  TEMP  for  resource  planning  are 
contained  in  Parts  IV  (Operational  Test  and  Evaluation  Outline) 
and  Part  V  (Test  and  Evaluation  Resource  Summary) .  The 
operational  tester  and  evaluator  participate  in  TEMP  development 
as  members  of  the  Test  Integration  Work  Group  (TIWG) ,  a  PM 
chaired  forum  promulgated  to  develop  the  TEMP  as  well  as  work 
issues  throughout  the  T&E  process. 

b.  The  TEMP  sets  in  motion  all  T&E  activities.  An  Army 
approved  TEMP  is  required  before  resources  can  be  documented, 
approved  and  tasked  for  in  the  Army's  Five  Year  Test  Program 
(FYTP) .  The  FYTP  seeks  to  identify  all  resources  required  NLT 
two  years  out  from  T-date  (the  date  record  testing  begins) .  This 
is  a  must  to  align  the  user  unit  training  and  readiness 
objectives  with  those  of  OT  in  a  mutually  supporting  manner 
without  degrading  the  readiness  of  units  tasked  to  support  OT. 

c.  Every  effort  must  be  made  to  identify  long  lead  resource 
requirements  such  as,  instrumentation,  targets,  modeling  and 
simulation,  and  other  budgeted  items.  Accordingly,  these  will  be 
reflected  in  command  budgets  and  outline  test  plans  (OTPs)  so 
they  can  be  developed  and  available  in  a  timely  manner. 

d.  TEMPS  should  not  identify  a  specific  test  unit  or  location 
unless  only  one  option  is  feasible.  The  Army's  Test  Schedule  and 
Review  Committee  (TSARC)  takes  all  requirements  into  account  when 
developing  who  will  support  and  where  OT  will  be  accomplished  in 
order  to  promote  equity  among  support  providers,  and  support 
other  OT  principles. 

e.  TEMPS  must  identify  all  OT  to  be  accomplished  before 
support  can  be  approved.  This  includes  Developmental  Tests  (DT) 
requiring  user  support.  Early  User  Tests  and  Experimentation 
(EUT&E) ,  Force  Development  Test  and  Experimentation  (FDT&E) , 
Limited  User  Tests  (LUT) ,  Initial  Operational  Test  and  Evaluation 
(lOT&E) ,  and  Follow-on  Test  and  Evaluation  (FOT&E) .  All  these 
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categories  of  tests  must  be  planned  and  coordinated  in  the  Army's 
TSARC  (see  figure  4-3)  and  FYTP  processes. 

4-5.  TSARC/FYTP  Processes 

a.  The  TSARC  is  a  General  Officer  Committee,  chaired  by 
Commander,  OPTEC,  that  meets  semiannually  to  provide  recommended 
priorities  and  resource  support  responsibilities  for  user 
supported  tests  to  the  DA  DCSOPS  for  approval  and  implementation. 
The  end  products  of  the  TSARC  are  the  FYTP,  and  test  priority 
lists  for  the  current  and  budget  year. 

b.  Resource  support  responsibilities  are  provided  in  detail 
in  outline  test  plans  (OTP)  submitted  to  the  TSARC  by  operational 
and  developmental  testers.  (See  figure  4-1,  OTP  Format.) 

(1)  All  direct  costs  for  operational  testing  are  delineated 
in  an  OTP.  It  lists  the  necessary  resources  and  the 
administrative  requirements  to  support  an  operational  test  and 
evaluation,  as  well  as  associated  suspense  dates  and  test 
milestones. 

(2)  When  included  in  the  approved  FYTP,  an  OTP  (see  figure 
4-1)  becomes  a  formal  resource  tasking  document  for  test 
execution  and  resource  allocation  within  program  and  budget 
constraints. 

(3)  OTPs  are  prepared  by  the  operational  tester  as 
designated  by  HQDA  DCSOPS  (or  materiel  developer  for  TT  when 
non-organic  or  user  troops  are  required)  and  maintained  by 
Headquarters,  OPTEC  for  the  TSARC  process.  OPTEC  is  the 
operational  tester  for  most  Army  Acquisition  Category  and  DOT&E 
oversight  program  tests.  However,  USAISC,  USAHSC,  USAINSCOM,  and 
others  are  designated  as  the  operational  tester  for  spec  ic 
programs . 

(4)  Preparation  of  the  OTP  begins  following  approval  of  the 
requirements  document  and  a  request  from  the  Project 
Manager/Program  Executive  Officer  to  OPTEC  for  evaluator  and 
tester  members  for  the  TWIG.  OPTEC  establishes  OTP  milestones 
concurrent  with  the  assignment  of  testers  and  evaluators.  Final 
TSARC  approval  of  the  OTP  should  take  place  no  later  than  24 
months  before  test  execution  and  in  no  case  less  than  one  year 
prior  to  execution.  These  milestones  are  critical  to  align 
testing  and  unit  training  objectives  and  minimize  adverse  effects 
of  testing  on  user  test  unit  and  personnel  readiness. 
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(5)  Test  requirements  that  do  not  allow  the  one-year 
^cation  can  only  be  approved  on  an  exception  basis  by 

submitting  a  proposed  OTP  to  the  Chairman  of  the  TSARC  (OPTEC 
Commander)  for  *'Out-of -Cycle"  coordination  by  the  TSARC  members 
and  subsequent  approval  by  DA  DCSOPS.  Such  a  submission  can  only 
be  submitted  by  a  memo  of  transmittal,  signed  by  a  General 
Officer  TSARC  member. 

(6)  No  OTP  will  approved  without  an  Army  approved  TEMP. 

c.  The  TSARC  establishes  priorities  among  the  tests,  resolves 
resource  issues  and  conflicts,  and  presents  a  prioritized  package 
of  OTPs  to  the  DA  DCSOPS  for  approval.  Once  approved  the 
compendium  of  OTPs  are  taskers  for  test  support  and  collectively 
are  known  as  the  Army’s  current  FYTP.  The  priority  lists  become 
guidelines  whereby  supporting  commands  apply  limited  resources  in 
rank  order.  The  approved  FYTP  is  published  and  distributed  by 
OPTEC  semiannually. 

d.  The  TSARC  is  supported  by  two  working  group  sessions  that 
introduce  new  requirements,  revise  current  plans  as  needed,  and 
develop  and  work  on  test  support  issues.  These  Initial  TSARC 
Working  Group  and  Mid-Cycle  Working  Group  sessions  are  chaired  by 
the  DCSOPS  of  OPTEC.  (See  figure  4-2,  TSARC  Process.)  Detailed 
procedures  of  the  working  groups  and  the  General  Officer  TSARC 
are  provided  in  the  TSARC  Handbook  published  by  OPTEC.  The 
charter,  scope,  membership  and  responsibilities  of  the  TSARC  are 
provided  in  AR  15-38.  (See  figure  4-3,  TSARC  Membership.) 

e.  All  the  types  of  test  mentioned  in  the  TEMP  paragraph 
above  will  be  included  in  FYTP.  This  includes  any  developmental 
or  operational  tests  that  require  test  support  assets  outside  the 
OTP  proponent  command  and  included  in  the  generic  definition  of 
OT.  These  include  all  of  the  categories  discussed  below. 

(1)  Developmental  Test  (DT) .  The  majority  of  TT  are  not 
included  in  the  TSARC/ FYTP  processes  as  they  are  conducted  solely 
from  within  the  resources  of  the  Materiel  Developer.  However, 
for  those  that  require  resources  not  controlled  by  the  Materiel 
Developer  (e.g.,  user  troops,  and  non-materiel  developer 
installations  or  facilities,  who's  use  compete  against  other  FYTP 
requirements) ,  must  be  submitted  for  approval  and  prioritization 
into  the  FYTP.  AMC  representatives  normally  submit  and 
coordinate  these  requirements  in  the  TSARC  process  and  provide 
funding  to  supporting  commands  as  required. 

(2)  FDTE.  These  are  conducted  early  in  the  acquisition 
process  for  materiel  systems,  and  for  non-materiel  acquisition, 
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as  desired  by  the  combat  and  training  developers  to  examine  the 
effectiveness  of  existing  or  proposed  concepts  of  training, 
logistics,  doctrine,  organization,  and  materiel.  FDTE  may  be 
used  to  determine  essential  and  desirable  capabilities  or 
characteristics  of  proposed  systems  or  can  assist  in  refining 
concepts  of  employment,  logistics,  training,  organization,  or 
personnel.  The  CBTDEV  or  TNG  DEV  is  the  proponent  for  such 
tests.  Accordingly,  TRADOC  introduces  these  types  of  tests  into 
the  TSARC  process  and  provides  funding  support  for  the  test  as 
required.  OPTEC  normally  conducts  these  tests  providing 
evaluation  support  as  agreed  to  in  a  mutual  memorandum  of 
understanding  with  TRADOC. 

(3)  EUTE.  This  is  a  generic  type  of  operational  test 
encompassing  all  system  focus  tests  employing  representative  user 
troops  during  the  demonstration  validation  phase  prior  to^ 
Milestone  II.  The  purpose  of  the  EUTE  is  to  test  a  materiel 
concept,  support  planning  for  training  and  logistics,  identify 
inter-operability  problems,  and/or  determine  future  testing 
requirements.  The  TIWG  must  identify  the  test  need  in  the  TEMP 
and  OPTEC  is  responsible  for  introducing  the  OTP  into  the  TSARC 
process,  RDTE  funding  for  test  costs  (see  Figure  4-2),  and 
conduct  of  the  test  and  evaluation. 

(4)  LUT.  The  LUT  is  a  generic  term  encompassing  all  RDTE 
funded  operational  testing  between  Milestone  II  and  Milestone  III 
that  is  not  a  part  of  lOTE.  These  test  address  limited  user  and 
operational  issues  and  are  normally  limited  to  a  single  issue. 

The  TIWG  must  identify  the  test  need  in  the  TEMP  and  OPTEC  is 
responsible  for  introducing  the  OTP  into  the  TSARC  process,  RDTE 
funding  for  test  costs  (see  figure  4—2) ,  and  conduct  of  the  test 
and  evaluation. 

(5)  lOT.  The  lOT  is  a  field  test,  under  realistic 
operational  conditions,  of  a  prodv  ion-representative  system  for 
determining  its  operational  effect  ness  and  suitability  for  use 
by  typical  users  in  combat  or  when  otherwise  deployed.  The  TIWG 
must  identify  the  test  need  in  the  TEMP  and  OPTEC  is  responsible 
for  introducing  the  OTP  into  the  TSARC  process,  RDTE  funding  for 
test  costs  (see  figure  4-2),  and  conduct  of  the  test  and 
evaluation. 

(6)  FOT.  These  tests  are  conducted  subsequent  to  a 
decision  to  procure  an  emerging  combat  system  to  evaluate 
required  fixes  noted  in  lOT  or  CE.  The  operational  evaluator 
determines  the  need  for  such  test  and  OPTEC  is  responsible  for 
introducing  the  OTP  into  the  TSARC  process,  OMA  funding  for  test 
costs  (see  figure  4-2),  and  conducting  and  evaluating  the  test. 
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f.  FYTP  Prioritization  Process.  Along  with  the  proposed 
FYTP,  the  TSARC  provides  priority  lists  for  current  and  budget 
fiscal  years  operational  testing.  The  lists  provide  Army 
guidance  for  the  relative  priorities  of  tests  for  resources. 

(1)  OPTEC  develops  draft  priority  lists  prior  to  the  TSARC 
Mid-Cycle  Working  Group  meeting.  The  drafts  contain  all  tests 
submitted  to  the  Initial  TSARC  Working  Group  for  the  current  and 
budget  fiscal  years.  DA  DCSOPS  provides  guidance  concerning 
prioritization  focusing  on  emerging  systems  designed  to  overcome 
critical  Army  battlefield  deficiencies. 

(2)  TSARC  members  provide  input  and  final  draft  priority 
lists  are  developed  at  the  Mid-Term  TSARC  Working  Group  meeting. 
OPTEC  makes  final  coordination  and  final  drafts  are  presented  for 
approval  at  the  General  Officer  TSARC  meeting.  Subsequent  to  DA 
DCSOPS  approval,  the  lists  are  published  by  OPTEC  along  with  the 
new  FYTP. 

(3)  Priority  lists  are  then  used  by  test  support  providers 
as  general  guidance  for  the  applications  of  limited  resources. 
Once  approved  the  Army's  FYTP  and  the  OTPs  contained  in  it, 
become  taskers  for  testing  support.  Because  tests  occur  at 
different  times  and  in  some  cases  there  are  more  demands  than 
resources,  resource  providers  must  manage  meticulously. 
Insupportable  taskings  should  be  brought  to  the  attention  of  DA 
DCSOPS  (DAMO-FDT)  as  soon  as  they  are  discovered. 

g.  Operational  Test  Readiness  Reviews  (OTRR) .  FYTP  tests  are 
scheduled  for  review  by  the  Commander,  OPTEC  or  other  OT&E 
activities  to  identify  problems,  make  changes  in  test 
preparations  as  required,  and  culminate  in  a  go/no  go  decision  by 
the  Commander,  OPTEC  or  his  designated  representative  as  to  the 
systems  readiness  for  test.  OTRR  procedures  are  detailed  in 
chapter  8. 

h.  Test  and  Evaluation  Principles.  To  maximize  the  return  on 
resources  committed  to  test  and  evaluation,  promote  continuity 
and  mutual  benefits  between  testing  and  training,  and  minimize 
the  impact  on  readiness  of  user  troops  who  participate  in  OT&E, 
the  TSARC/ FYTP  processes  seek  to  adhere  to  the  following 
principles. 

(1)  Minimize  the  effects  of  operational  testing  on  units; 
correlate  testing  and  training  management  processes;  assure  at 
least  one  year  notification  to  units  involved  in  testing;  and 
test  where  test  units  are  stationed. 
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(2)  Test  with  the  first  units  equipped  (FUE) :  this  allows 
the  practical  benefits  of  training  on  new  equipment  by  those  who 
will  first  have  to  use  it;  and  enhances  the  readiness  of  the  unit 
with  the  introduction  of  the  emerging  combat  system. 

(3)  Test  smart  by  using  simulations  and  models  where 

feasible:  these  can  save  troops  required,  time  required,  allows 

interactive  test  and  evaluation,  and  allows  test  and  evaluation 
of  concepts  not  possible  with  the  current  force. 

(4)  Combine  tests  where  feasible:  combining  FDT&Es  and 
lOT&Es,  and  TTs  and  operational  tests  have  proven  to  save 
resources  and  expedite  the  T&E  process. 

(5)  Use  the  minimum  essential  resources  to  achieve 
adequacy,  quality  and  credibility:  test  against  COIC  challenging 
additions  at  every  opportunity. 

(6)  Maintain  the  independence  of  testers  and  evaluators: 
conduct  OT&E  in  cooperation  with  the  materiel  and  combat 
developers,  but  separate  in  terms  of  control  of  OT&E  resources 
and  participation  in  OT&E  efforts. 

4-6.  Concept  Evaluation  Program  (CEP)  and  Customer  Test  (CT) 
Support 

In  many  instances,  combat  and  materiel  developers  have  need  for 
test  support  to  evaluate  concepts  or  innovations  prior  to  a 
materiel  solution  to  modernization  needs  being  developed.  OPTEC 
(TEXCOM)  and  AMC  (TECOM)  support  these  needs  categorized  as  CEP 
tests  or  CTs  on  a,  as  feasible  and  reimbursable,  basis.  Because 
they  are  usually  small,  short  notice,  and  do  not  compete  with 
FYTP  resources,  CEP  and  CT  coordination  and  support  are  planned 
and  executed  outside  the  FYTP/TSARC  processes. 

a.  CEP  The  CEP  is  a  TRADOC  program  to  provide  quick 
reaction  ar.T  innovative  evaluation  to  resolve  combat  and  training 
development  issues. 

(1)  A  TRADOC  CEP  Schedule  and  Review  Committee  (CEPSARC) , 
similar  to  the  TSARC  in  function,  meets  semiannually  to  approve 
and  prioritize  CEP  requirements.  Priorities  are  then  provided  to 
supporting  commands  for  tester  assignment. 

(2)  Because  of  the  nature  of  CEPs,  direct  coordination  with 
the  tester  headquarters  is  encouraged.  For  OPTEC  supported  CEP 
tests,  a  resume  sheet  delineating  support  requirements  (see 
figure  4-4)  is  required. 
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p)  TRADOC  is  responsible  for  funding  support  of  CEP  tests 
which  involve  RDTE  funds  specifically  programmed  and  budgeted  for 
CEPS  by  DA. 


b.  CT.  Customer  tests  is  a  generic  category  to  accommodate 
all  other  operational  testing  needs  not  provided  for  in  the  FYTP 
or  CEP  processes.  The  basis  of  test  support  and  coordination  is 
the  same  as  for  CEP  tests.  The  only  difference  is  in  tester 
priority,  wherein  CTs  are  lower  priority  than  CEP  tests  and 
therefore,  if  there  is  a  test  resource  conflict,  CTs  must  be 
coordinated  around  other  requirements. 

c.  To  facilitate  test  and  evaluation  support  for  CEP  and  Cts, 
OPTEC  has  established  Test  and  Evaluation  Coordination  Offices 
(TECO)  at  TRADOC  locations  (i.e..  Fort  Banning,  Fort  Knox,  Fort 
Lee,  Fort  Leonard  Wood,  Fort  Rucker,  and  Fort  Gordon) .  The  TECOs 
along  with  TEXCOM  Headquarters  at  Fort  Hood,  and  other  OPTEC  test 
directorates  at  Fort  Bragg,  Fort  Huachuca,  Fort  Sill,  and  Fort 
Hunter  Liggett  provide  informal  assistance  for  customer  test 
planning  and  coordination.  Submittal  of  formal  requests  for 
support  are  required  to  HQ  OPTEC  (ODCSOPS-TMD)  so  overall 
requirements  can  be  taken  into  consideration  and  formal  internal 
taskings  conducted. 

4-7.  Instrumentation 

Management  of  user  test  instrumentation  is  accomplished  through 
the  User  Test  Instrumentation  Program  (UTIP) .  The  UTIP  describes 
consolidates  and  priorities  instrumentation  projects.  It  also 
documents  OPTEC  command  decisions  relating  to  funding  levels  and 
priorities.  UTIP  projects  are  classified  as  major,  non-major, 
and  sustaining  instrumentation. 

a.  Major  instrumentation  is  defined  as  instrumentation  that 
has  a  joint  service  application,  multiple  command  requirement, 
requires  high  visibility,  or  a  total  life  cycle  cost  of  $5M  and 
above  (present  year  dollars) .  These  systems  typically  provide 
new  capabilities  or  significant  upgrades  to  existing 
capabilities. 

b.  Non-major  instrumentation  is  defined  as  instrumentation 
having  a  total  life  cycle  cost  of  less  that  $5M  and  is  not 
classified  as  major.  These  systems  typically  include  test 
specific  instrumentation  and  upgrades  or  enhancements  to  existing 
assets.  Sustaining  instrumentation  is  defined  as  repair, 
maintenance,  or  replacement  of  current  inventory.  Sustaining 
instrumentation  costs  will  not  exceed  $15K  Research,  Development, 
Test  and  Evaluation  (RDTE)  or  $100K  combined  RDTE  and  Other 
Procurement,  Army  (OPA) . 
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c.  To  acquire  new  instrumentation,  requirements  are  presented 
at  either  of  the  semiannual  Instrumentation  Requirements 
Conferences.  Instrumentation  requirements  conferences  are  hosted 
semiannually  in  May  and  November  each  year  to  prepare  the  UTIP 
for  execution  and  for  entry  into  the  POM  cycle.  Topics  of 
discussion  will  include  (but  not  limited  to)  policy /procedural 
changes,  current  funding,  fiscal  planning  guidelines,  and 
anticipated  test  datd  requirements  that  may  significantly  impact 
present  or  future  instrumentation  development  programs.  This 
conference  provides  a  forum  for  testers  to  propose 
instrumentation  development  projects  to  satisfy  its  future  test 
needs  and  to  provide  a  peer  review  of  other  candidate  projects. 

All  submissions  will  contain  a  need  statement,  instrumentation 
description,  estimated  funding  requirements  and  justification. 
Projects  will  be  reviewed  for  results  of  the  economic  analysis, 
unnecessary  duplication,  commonality,  interoperability  and 
compatibility  with  the  instrumentation  inventory  data  base. 
Conference  results  will  be  provided  to  OPTEC. 

d.  OPTEC  Command  Review  Committee  is  chaired  by  OPTEC  and 
composed  of  Colonel  (06)  level  representation,  to  review  and  to 
assure  that  instrumentation  programs  support  long  term  OT&E  plans 
and  are  in  concert  with  TSARC  test  priority  guidance.  The  review 
committee  will  either  approve  projects  for  funding  consideration 
or  return  it  to  proponent  for  resubmission  (if  applicable)  at 
next  cycle.  Committee  will  determine  funding  level  for  each 
approved  project  and  its  priority.  The  committee's  decisions 
will  be  documented  and  distributed  as  minutes  of  the  review 
committee  and  will  serve  as  the  basis  for  the  formulation  of  the 
UTIP.  Upon  approval,  the  UTIP  becomes  the  OPTEC  instrumentation 
master  plan. 

e.  OPTEC  will  provide  the  approved  UTIP  to  the  Program 
Manager  for  Instrumentation,  Targets  and  Threat  Simulators  (PM— 
ITTS)  for  review.  Major  instrumentation  projects  will  be 
prioritized  and  presented  by  PM-ITTS  to  the  General  Officer 
Steering  Council  for  ITTS  approval,  funding  and  program 
execution.  OPTEC  will  appoint  a  user  representative  to  monitor 
project  activity  and  contract  performance.  When  required,  a 
Users  Group  will  be  established  with  chair  designated  by  OPTEC  to 
elicit  changes  and  provide  recommended  program  changes  to  OPTEC 
command . 

4-8.  OT&E  Funding 

This  section  delineates  funding  processes  and  responsibilities  of 
OT&E.  Generally,  OPTEC  plans,  budgets,  and  programs  for  the 
majority  of  FYTP  execution  less  FDTE  (TRADOC) ,  TT  (AMC)  and 
selected  other  command  sponsored  testsj  TRADOC  is  responsible  for 
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CEP  tests;  and  the  customer  bears  responsibility  for  CT.  A  quick 
reference  to  test  and  evaluation  funding  by  type  T&E  activity, 
appropriation,  and  command  responsibility  is  provided  in  figure 
4-5.  Generally,  operational  test,  test  support  items  and  CE 
activities  are  funded  as  follows: 

a.  Operational  Tests.  lOT&E  on  a  system  is  usually  funded 
from  RDTE  appropriations.  Category  6.5  funds.  FOT&E  is  normally 
funded  from  OMA  funds  (AR  70-10) . 

b.  ^  Continuous  Evaluation  (CE) .  CE,  to  include  analysis, 
modeling,  and  simulation  proposed  by  the  evaluator,  is  funded 
from  OMA  funds. 

c.  Concept  Evaluation  Program  (CEP).  Testing  for  concept 
evaluation  is  done  using  RDTE  funds  for  materiel  development  and 
with  OMA  funds  for  combat  developments  (AR  70-10) . 

d.  Multiservice  tests.  •  Costs  for  Army  related  portions  of 
nwilti— service  tests  are  funded  from  the  RDTE  appropriation  if 
testing  occurs  before  the  decision  to  enter  full  production. 

Costs  for  testing  after  the  full  production  decision  are  funded 
from  the  OMA  appropriation. 

e.  Special  tests.  Special  operational  tests,  such  as  special 
access  program  tests,  may  be  specifically  requested  by  DA  or 
higher  organizations.  Such  tests  would  not  normally  be  included 
in  initial  TEMP  planning  nor  in  an  OTP.  In  such  an  instance,  the 
tester  will  require  identification  of  funding  sources  from  the 
test  requester. 

f.  Customer  tests.  Normally  combat  developers,  but  others  as 
well,  may  request  OPTEC  or  TECOM  support  limited,  short-notice 
tests  for  any  reason.  These  assume  that  all  external  support 
requirements  will  be  coordinated  by  the  customer  and  the  variable 
test  costs  (see  figure  4-2)  vill  be  reimbursed  by  the  customer. 

g.  Ammunition.  Consumable  rounds  of  standard  ammunition  and 
tactical  missiles  required  in  support  of  operational  testing  are 
provided  by  the  procurement  appropriation  or  from  existing 
inventory,  on  a  priority  basis  without  reimbursement.  For  all 
short-supply  conventional  ammunition,  the  Committee  for 
Ammunition  Logistics  Support  (CALS)  allocates  and  prioritizes  all 
ammunition  and  pyrotechnics  including  test  requirements.  As  a 
user  test  representative,  OPTEC  is  a  member  of  the  CALS. 
Developmental  ammunition  and  missiles  required  to  support 
operational  testing  are  identified  early  in  the  program  and 
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financed  through  the  RDTE  appropriation  of  the  system's  materiel 
developer. 


h.  User  Test  Flying  Hour  Program.  OPTEC  manages  this  program 
which  allocates  flying  hours  to  test  agencies  in  support  of 
operational  testing.  The  basis  for  flying  hour  allocations  is 
the  flying  hour  requirements  reflected  in  OTPs  in  approved  FYTP. 
OPTEC  submits  neports  as  outlined  in  AR  95-33,  Army  Aircraft 
Inventory,  Status,  Flying  Time,  and  the  TSARC  Handbook. 


i.  Integrated  Logistics  Support  Plan  (ILSP) .  The  ILSP  is 
prepared,  coordinated,  and  approved  by  the  Materiel  Developer. 
Resource  requirements  for  operational  testing  of  logistics  ^ 
supportability  are  incorporated  into  the  OTP  by  the  operational 
testing  activity.  The  RDTE  funds  required  to  support  ILS  test 
plans  are  listed  in  the  Support  Resource  Funds  section  of  the 
ILSP. 


i.  OT&E  Inputs  to  the  Planning,  Programming,  Budgeting,  and 
Execution  System  (PPBES) .  Inputs  to  the  PPBES  from  OT&E  go  to  DA 
in  one  of  two  submissions  from  the  command;  The  Program  Analysis 
Resource  Review  (PARR)  and  the  Command  Operating  Budget  (COB). 
Program  Decision  Increment  Packages  (PDIP)  describing  operational 
tester  requirements  for  RDTE  appropriations  are  extracted  from 
the  PARR  at  DA  level  and  input  to  the  Long-Range,  Research, 
Development  and  Acquisition  Plan  (LRRDAP) .  The  LRRDAP  and  the 
FYTP  are  inputs  to  ASA(RDA)  guidance  on  COB  preparation  by  the 
OT&E  activity.  OT&E  is  costly  and  every  effort  must  be  made  to 
conserve . 


(1)  Funding  requirements  must  be  identified  as  far  forward 
as  possible.  The  TSARC/ FYTP  processes  are  dynamic  in  that  OTP 
are  modified  in  each  cycle  to  reflect  funding  refinements. 
Although  the  tester  has  principal  responsibi  ty  for  documenting 
requirements  in  OTP,  many  others  must  provi  Input  to  capture 
all  costs  as  quickly  as  possible. 

(a)  Overall  programming  and  budgeting  is  accomplished  by 
ASA(RDA)  and  DA  DCSCOPS  required  to  conduct  OT.  ASA(RDA) 
provides  RDTE  funding,  while  DA  DCSOPS  provides  OMA  funding. 

Both  of  these  appropriations  are  used  to  support  OT  depending  on 
the  phase  of  development  in  which  testing  is  performed.  If 
testing  is  performed  in  the  development  stage,  then  it  is  funded 
through  RDTE  appropriation;  however,  if  testing  occurs  in  the 
production  phase,  the  funding  is  OMA. 

(b)  The  TIWG  must  identify  types  of  test  to  be 
conducted;  and  the  long  lead  items  (e.g.,  instrumentation,  threat 
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systems, targets,  simulators,  modeling),  size  and  type  of  test 
units  involved,  test  articles,  support  equipment,  training 
requirements,  and  estimated  funding  required  for  each  in  the 
TEMP.  Remember  that  OTP  without  Army  approved  TEMP  are  withdrawn 
from  the  FYTP  by  the  TSARC  as  not  executable. 

(c)  The  Combat  Developer's  COIC  drive  test  requirements 
more  than  any  other  element  of  OT6E.  The  number  of  tests,  the  of 
event  repetitions,  length,  etc.,  are  contingent  on  the  COIC  and 
the  AOIC  developed  by  the  operational  evaluator.  Expeditious 
approval  of  both  the  COIC/AOIC  are  essential  to  resource  planning 
and  programming. 

(d)  DOTE  and  DUSA(OR)  review  and  approval  of  test  and 
evaluation  concepts  for  major  and  oversight  systems  in  many 
instances  add  significant  resource  requirements  to  scheduled 
tests  late  in  the  OTP  development  process.  These  unprogrammed 
addi'tions  must  receive  the  closest  scrutiny  and  if  warranted  be 
provided  to  the  operational  tester  at  the  earliest  possible  time. 

(e)  The  major  cause  of  test  cost  increases  and  negative 
impacts  on  user  units  involved  in  operational  testing  is  late 
realization  that  test  articles  will  not  be  ready.  It  is 
incumbent  on  the  PM/PEO  to  notify  the  TiE  community  at  the 
earliest  possible  time  that  a  system  will  not  be  ready  to  go  to 
test.  Any  test  slip  costs  dollars,  and  the  closer  to  the  test 
date  and  longer  the  slip,  the  more  it  costs.  Test  unit  training 
schedules  become  disrupted  and  therefore  their  readiness  with 
unplanned  requirements  to  support  OT&E.  The  more  advanced  notice 
received  of  an  impending  slip  in  availability  of  test  articles, 
the  less  the  impact  on  costs  and  test  units. 


OTEMS  VERSION 


CLASSIFICATION 

OUTLINE  TEST  PLAN  (OTP) 


TYPE  SUBMISSION 
DATE 


TESTER:  OFFICE:  DSN: 

EVALUATOR:  OFFICE:  DSN: 

TEST  title- 
test  TYPE: 

ACQUISITION  MILESTONE  SUPPORTED: 

ACQUISITION  CAT:  _  DECISION  REVIEW:  _  DOD  OVERSIGH":  - 

TEST  AUTHORITY:  _  T&E  MGMT:  _ 

COMBAT  DEVELOPER: 

MATERIEL  DEVELOPER: 

DA  STAFF  PROPONENT: 

TEST  ORGANIZATION: 

TEST  UNIT: 

TES"  LOCATION: 

TEST  DATES:  START:  _  COMPLETION:  _  RESOURCE:  - 

AMMO:  _  AVNHRS:  _  CONTR:  _  INST:  _  SiM:  _  TG".  -  TGT  SiM - 

TOTAL  DIRECT  TEST  AND  EVALUATION  COSTS  (IN  THOUSANDS): 

PROG  BY  APPN  FY _  FY _  FY -  FY - 

_  _  0.0  0.0  0.0  0.0 

1.  REFERENCES: 

A.  Requlrenen:/Tasking  Documen;(s). 

B,  TEMP. 

C  Waivers. 

D  Previous  Testing 
E.  Planned  Future  Tests. 

2  PURPOSE/FUNCTIONAL  DESCRIPTION. 

A.  Purpose. 

B.  Functional  Description. 

3  CRITICAL  OPERATIONAL  ISSUES  (COi) 

4.  SCOPE  AND  tactical  CONTEX* 

A.  Scope. 

B.  Tactical  Context. 

5.  IMPACT  STATEMENTS. 

A.  Environmental,  Laser  and  Energy  implications. 

B.  Training  implications. 

C.  SiGSEC/OPSEC  implications. 

D.  Human  Volunteers. 

E.  Radionuclides  Certification. 


XXXX-XX-XXXX-XXXXA-  1 
(CLASSIFICATION) 

FIGURE  A*1 


CLASSIFICATION 

6.  other  resources  required. 

A.  Suppor:  Packages. 

B.  Sysien  Safety  (Requirements/Releases). 

C.  Photographic  Support. 

D.  Contractor  Stuoies  or  Support. 

E.  Meteorological  Support. 

F.  Security  Requirements. 

G  Other 

7.  PQ  N~S  OF  CON’AC” 

SECTION  I 

~EST  RESOURCE  REQUIREMENTS 

1.  TEST  DIRECTORATE. 

A.  Personnel  Requirements. 

B.  Equipment  Requirements. 

2.  PLAYER  PARTICIPANTS. 

A.  Personnel  Requirements. 

(1)  Individual  Personnel. 

(2)  Unit/Element  Personnel. 

B  Equipment  Requirements. 

3.  ITEM(5)  TO  BE  TES’ED. 

A.  Test  Items. 

E  Support  Requirements 
4  DA~A  COLLECTiON/ADP  SUPPORT 

A  Data  Collection/Processing  System. 

B,  ADP  Facility  Support. 

C  Contractor  or  Other  Government  Agencies. 

5.  A.MMUNIT1ON,  M:S5  LES,  and  pyrotechnics. 

A.  Ammunition  and  Pyrotechnics. 

B  Missiles. 

6  PETROLEUM,  0  LS,  AND  LUBRlCAN'S  (POL)  SUPPLIES 
7.  ,NSTRUMEN"AT:0N 
A  Equipment. 

B  Contractor  or  Other  Government  Agencies 
8  "EST  FAC  LiTlES/  NSTALLATlON  SUPPORT. 

A  Test  Facility  Range  Support. 

B.  Communication/Engineering  Support. 

C.  Installation  Support. 

D.  Other  Support. 

9.  THREAT  SIMULATORS/OTHER  SIMULATORS/TARGET  VEHICLES. 

A.  Threat  Simulators. 

B  Other  Simulators. 

C  Target  Vehicles* 

10.  FLYING  HOURS  SUPPORT. 


XXXX-XX-XXXX-XXXXA-  1 
(CLASSIFICATION) 


Figure  4-1.  (cont) 
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CLASSIFICATION 


SECTION  II 

EVALUATION  RESOURCE  REOUiREMEN"S 


1.  PERSONNEL  REOU  REMENTS. 

2.  EOUlPliENT  REQUiREnENTS. 

3.  ADP  SUPPORT  REOUlREMENTS. 

4.  INSTRUMENTATION  REQUIREMENTS. 

5.  FACILITIES  SUPPORT  REOU'REMENTS. 


SECTION  III 

TEST  MILESTONES/CONTRACT  SUMMARY 


1.  MILESTONES. 

2.  CONTRACT  REOU'REMENTS  SUMMARY. 


XXXX-XX-XXXX-XXXXA-  I 
(CLASSIFICATION) 


Figure  4-1 .  (cont) 


CLASSIFICATION 
SECTION  IV 

TEST  COST  ESTIMATES 

1.  EVALUATION  COSTS  ESTIMATES  (IN  THOUSANDS). 

TEST  NUMBER:  XXXX-XX-XXXX-XXXXA  DATE  PREPARED:  DD  MMM  YYYY 

TEST  TITLE:  XXXXXXXXXXXXXXXXXXXXX 
TEST  TYPE:  XXXXXXXXXXXXXXXXXXXXX 


CATE(50RY  OF  COST 

PROG  BY 

APPN 

FY  FY. 

FY. 

FY. 

(a)  CIVILIAN  HIRE  (CIV  PAY) 

XXXX 

XXXX 

0.0 

0.0 

00 

0  0 

(b)  CIVILIAN  overtime 

XXXX 

XXXX 

0.0 

0.0 

00 

00 

(C)  TDY  (EVALUA'.ON) 

XXXX 

XXXX 

0.0 

0.0 

0.0 

0  0 

(e)  LEASE/RENT AL-C0MM0/U"!L' 

XXXX 

XXXX 

0.0 

0  0 

0.0 

0  0 

C) CONTRACTS 

XXXX 

XXXX 

0.0 

0.0 

0  0 

00 

(h)  SUP?L'ES/MA"ERiEL. 

XXXX 

XXXX 

00 

0  0 

00 

0  0 

(i)  EOU  PMENT 

XXXX 

XXXX 

0.0 

0.0 

00 

0  C' 

(j)  INSTRUMEN~A“:DN 

XXXX 

XXXX 

0.0 

0,0 

00 

0  - 

TOTAL  EVAL  COS’  PROG  BY 

XXXX 

XXXX 

0.0 

0.0 

0.0 

00 

TO’AL  EVALUAT  ON  COSTS 

0.0 

0  0 

00 

0  0 

XXXX-XX-XXXX-XXXXA-  I 
(CLASSIFICATION) 


Figure  4-2. 


To:  Pat  Mawby 


From:  MAJ  Kenneth  L.  Wilson 


11-30-35  4:07pni  p.  3  of  7 


CLASSIFICATION 

2.  DIRECT  TEST  COSTS  ES'iMA’ES  (IN  Thousands). 

TEST  NUMBER:  XXXX-XX-XXXX-XXXXA  DATE  PREPARED:  OD  MMM  YYYV 

TEST  TILE:  XXXXXXXXXXXXXXXXXXXXX 
TEST  TYPE:  XXXXXXXXXXXXXXXXXXXXX 

CATE(50RY  of  cost  prog  by  APPN  FY _  FY _  FY _  FY_ 


(a)  CIVILIAN  HIRE  (CIV  PAY) 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(b)  CIVILIAN  overtime 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(C)  TOY  (EVALUATION) 

XXXX 

xxxx 

0.0 

0.0 

0,0 

(d)  TRANSPORT  OF  TEST  ITEM 

XXXX 

xxxx 

00 

0.0 

0.0 

(e)  LEASE/RENT AL-C0MM0/U"IL 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(f)  CONTRACTS 

XXXX 

xxxx 

0,0 

0.0 

0.0 

(g)  POL 

XXXX 

xxxx 

0.0 

00 

0.0 

(b)  SUPPLiES/.MA'ERiEL 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(i)  EQUIPMENT 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(1)  INSTRUMENTATION 

XXXX 

xxxx 

0,0 

0.0 

0.0 

(k)  THREAT  Simulators 

XXXX 

xxxx 

0.0 

■00 

0,0 

(1)  OTHER  SIMULATORS 

XXXX 

xxxx 

0,0 

0.0 

0,0 

(n)  targets 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(n)  ARMY  AV:A~I0N  SUPPOP"  COS" 

XXXX 

xxxx 

0.0 

0.0 

0.0 

(0)  C"HER  SERVICES  D  RFC"  S?" 

XXXX 

xxxx 

0.0 

0  0 

0.0 

TO" AL  DIRECT  COST  PROG  BY 

XXXX 

xxxx 

0.0 

0.0 

0.0 

"OTAL  O  RECT  TEST  COSTS 

xxxx 

xxxx 

0,0 

00 

0.0 

"0"AL  EVAL  COST  PROG  BY 

XXXX 

xxxx 

0.0 

00 

0.0 

TOTAL  EVALUATION  C05"5 

xxxx 

xxxx 

0.0 

00 

0.0 

(p)  O'HER  SERVICES  5P~  COST 

xxxx 

xxxx 

00 

0  0 

0  0 

(q)  AMMUNITION  COS" 

xxxx 

xxxx 

0.0 

0  0 

0  0 

TOTAL  TEST  COSTS 

xxxx 

xxxx 

0.0 

0,0 

0.0 

Figure  4-2 i  (cent) 
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SEMI-ANNUAL 
TSARC  PROCESS 


CURRENT 


DA 

FYTP 


FUNDING  develop  PRIORITIES 


INSTRUMENTATION 
SCHEDULE  CONFLICTS 
PROGRAM  CONFLICTS 
SIMULATORS 
AMMUNITION 
FLYING  HOURS 
TARGETS 


(JUN/DEC) 

1  II 

NEW 

GO 

M 

DA 

TSARC 

n 

FYTP 

RESOLVES  ISSUES 
RECOMMENDS 
FYTP 

APPROVAL 
TEST  PRIORITIES 


TEST  SCHEDULE  AND 
REVIEW  COMMITTEE 

1 

CDR,  OPTEC  •  CHAIRMAN 

ODCSOPS 

ODISC4 

ODCSLOG 

ODCSPER 

ODCSINT 

OTSG 

ASA  (RDA) 

ASA  (FM) 

AMC 

TRADOC 

FORSCOM 

USAREUR 

USARPAC 

USASOC 

USAISC 

INSCOM 

TECOM,  TEXCOM.  ISEC,  HSC  •  MAY  PROVIDE  REPRESENTATIVES. 

OTHER  STAFF  AGENCIES/ARMY  COMMANDS  ARE  INVITED  WHEN  TEST  PROGRAMS 
FALL  IN  THEIR  FUNCTIONAL  AREA  OF  RESPONSIBILITY  OR  INVOLVE  THEIR  RESOURECES. 


OTEMS  VERSION 


CLASSIFICATION 

RESUME  SHEET  (RS) 


DATE 


TESTER: 


OFFICE; 


DSN: 


TEST  TITLE:  CRC:  _ 

TEST  TYPE: 

TEST  AUTHORITY:  _ 

COMBAT  DEVELOPER. 

TES"  ORGAN  1 2  AT  I  or; 

TEST  UNIT; 

TEST  LOCATION: 

TEST  DATES:  STAR":  _  COMPLETION:  _  RESOURCE.  - 

AMMO:  _  AVN  HRS:  _  CONTR:  _  INST;  _  SIM:  _  T6T:  _  TGT  SIM - 

TOTAL  DIRECT  TEST  AND  EVALUATION  COSTS  (IN  THOUSANDS): 

PROG  BY  APPN  FY _  FY _  FY _  FY - 

_  _  00  0.0  0.0  00 


"HESE  F  ELDS  ARE  EMPLOYED  FOR  CEPS  ONLY 


TEST  AUTHORIZATION 

TEST  TIMEFRAME 

PROPONEN"  PR,OR'"Y 

1.  _DA  DIRECTED/ON-GOING 

1,  _  WITHIN  1  OTR 

1  _  1 

2.  _TRADOC  DiREC’ED 

2  _  WITHIN  2  OTRS 

2.  _  2 

3.  _  MAA  DEFICIENCY 

3  _  WITHIN  3  OTRS 

3  _  3 

4  _BDP  DEFICIENCY 

4  _  W'lTHIN  4  OTRS 

4,  _  4 

5.  _MAT'L  ROM~/DO"L 

6.  _  NEW  IDEA 

5  _  OVER  1  YR 

5  _  O'HER 
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1.  REFERENCES; 

A.  Requirenen:/Tasklng  DocunentCs). 

B.  TEMP. 

C  Waivers 
D  Previous  Tes'.ing 
E.  Planned  Future  Tests 

2.  PURPOSE/FUNCT.ONAL  DESCRiP'iON 
A.  Purpose. 

B  Functional  Description 

3.  CRITICAL  OPERA'iONAL  ISSUES  (COU 
4  SCOPE  AND  tact  CAL  CON'EX^. 

A.  Scope. 

B.  Tactical  Context. 
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5.  IMPACT  5TATEMEN“S 

A.  Environmental,  Laser  and  Energy  implications 

B.  Training  implications. 

C  SiGSEC/OPSEC  Implications. 

D.  Human  Volunteers. 

E.  Radionuclides  Certification. 

6  OTHER  RESOURCES  REQUIRED. 

A  Support  Packages 

B.  System  Safety  (Requirements/Releases) 

C.  Photographic  Support. 

D.  Contractor  Studies  or  Support. 

E.  Meteorological  Support. 

F.  Security  Requirements. 

G.  Other. 

7.  POINTS  OF  CONTAC". 

SECTION  I 

"E5"  RESOURCE  REQUIRE.MENTS 

1  TEST  DIRECTORATE. 

A.  Personnel  Requirements 

B.  Equipment  Requirements 

2  PLAY'ER  PARTICI  PAN’S 

A  Personnel  Requirements 

(1)  individual  Personnel. 

(2)  Unit/Element  Perscnnel, 

B  Equipment  Requirements 

3.  ITEM(S)  TO  BE  TESTED. 

A.  Test  Items. 

B  Support  Requirements. 

4.  DATA  COLLECTION/ AD?  SUPPORT. 

A  Data  Collection/Processing  System. 

B  ADP  Facility  Support. 

C.  Contractor  or  Other  Governmeht  Agencies 

5.  AMMUNITION,  MiSS  lES,  AND  PYROTECHN  CS 

A.  Ammunition  anq  Pyrotechnics. 

B.  Missiles. 

6.  PETROLEUM,  OILS,  AND  LUBR'CANTS  (POL)  SUPPLIES 

7.  INSTRUMENTATION 

A.  Equipment. 

B.  Contractor  or  Other  Government  Agencies. 
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8  TEST  FACiLiTlES/iNSTALLATlON  SUPPORT. 

A.  Test  Facility  Range  Support. 

B.  Comnunication/Engineering  Support. 

C.  Installation  Support. 

D.  Other  Support. 

9.  THREAT  SIMULATORS/OTHER  S:T1ULAT0RS/TARGET  VEH  CLES 

A.  Threat  Simulators. 

B.  Other  Simulators 

C.  Target  vehicles. 

10.  FLYING  hours  SUPPORT. 
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1.  MILESTONES. 

2.  contract  RE0U.REMENT5  SUMMARY 
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I,  DIRECT  TEST  COSTS  EST  MATES  (iN  THOUSANDS). 

"ES"  NUMBER;  XXXX-CP-XXXX-XXXXA  DA'^E  PREPARED:  DD  MMM  YYYY 

TEST  TITLE  XXXXXXXXXXXXXXXXXXXXX 
TEST  TYPE:  XXXXXXXXXXXXXXXXXXXXX 


CATEGORY  OF  COST  PROG  BY 

(a)  CIVILIAN  hire' (CIV  PAY)  XXXX 

(b)  CIVILIAN  OVERTITIE  XXXX 

(C)  TDY  (EVALUATION)  XXXX 

(0)  TRANSPORT  OF  TEST  ITEM  XXXX 

(e)  LEASE/RENT AL-COMMO/U~lL  XXXX 

(f )  CONTRACTS  XXXX 

(g)  POL  .  XXXX 

(h)  SUPPL!ES/MA“ER  EL  XXXX 

(l)EOU‘PMENT  -XXXX 

(J)  INSTRUMEN'ATION  XXXX 

(k)  THREAT  SIMULATORS  XXXX 

(l)  other  SIMULATORS  XXXX 

(n)  targets  XXXX 

(n)  ARMY  AV'A“iON  SUPPOR"  COS'  XXXX 
(0)  O'HER  SERV  CES  D  REC"  SP"  XXXX 

tj'al  Direct  cos'  prog  by  xxxx 

TO'AL  DIRECT  TEST  COSTS  XXXX 

'D'AL  EVAL  COS'  PROG  BY  XXXX 

TOTAL  EVALUA'iON  COSTS  XXXX 

(p)  other  services  SPT  COS'  XXXX 

(q)  AMMUN.TiON  COST  XXXX 

TOTAL  TEST  COSTS  XXXX 
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Chapter  5 
Test  Execution 


Section  I 
General 


5-1.  Introduction  to  test  execution 

The  operational  tester  mission  is  to  plan,  execute,  and  report 
OT.  This  mission  is  accomplished  through  a  structured  approach 
to  developing  plans  and  procedures  to  meet  the  test  requirements. 
These  plans  and  procedures  are  identified  and  performed  lAW  a 
sequence  of  steps  dependent  upon  time  phasing  within  overall  test 
milestones.  Each  period  of  time  phasing,  or  a  test  phase, 
requires  specific  actions  to  be  accomplished  by  participants  and 
appropriate  test  documentation  to  be  produced  that  contains  the 
procedures  for  the  OTE  that  have  been  developed.  Each  of  these 
test  phases  address  a  specific  time  period  during  the  overall  OT 
process.  The  six  functional  phases  are: 

a.  Preliminary  Analysis  and  Planning  Phase.  See  Chapter  3. 

b.  Test  Design  Planning  Phase.  See  Chapter  3. 

c.  Detailed  Test  Planning  Phase.  See  below  and  Section  III. 

d.  Test  Execution  Phase.  See  below  and  Section  XV. 

e.  Data  Reduction  and  Analysis  Phase.  See  below  and  Sections 
•XVIII  through  XXIII. 

f.  Report  Writing  and  Publishing  Phase.  See  Chapter  6. 

5-2.  Detailed  test  planning  phase 

This  phase  of  the  planning  process  expands  and  finalizes  the 
detailed  procedures  that  will  be  used  during  the  test  execution 
and  reporting  phases.  Following  publication  of  the  approved  TEP , 
the  operational  tester,  in  conjunction  with  the  test  proponent 
and  other  appropriate  commands  and  agencies,  develops  the  DTP. 

The  DTP  translates,  as  required,  the  test  design  contained  in  the 
TEP  into  specific  instructions  for  the  actual  conduct  of  test. 
Test  control,  data  management,  training,  and  test  support 
requirements  and  procedures  are  finalized  in  the  DTP.  DTP  are 
for  internal  use  of  the  operational  tester,  but  may  be  provided 
to  interested  government  agencies  for  information. 

5-3.  Test  execution  phase 
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During  this  phase,  planned  events  are  conducted  and  data  are 
collected  and  recorded.  Deviations  from  planned  events  are 
documented.  The  products  of  this  phase  are  comprehensive  data 
obtained  lAW  the  planned  test  methodology  which  will  be  used  to 
address  the  test  issues. 

5-4.  Data  reduction  and  analysis  phase 

Data  reduction  includes  the  transfer  of  data  from  raw  form  to 
data  bases  or  the  processing  of  data  collected  with  test 
instrumentation  lAW  preplanned  procedures  and  methodology.  Data 
reduction  may  begin  during  the  test  execution  phase  and  results 
in  an  authenticated  level  3  data  base  to  meet  analysis 
requirements.  This  data  base  will  be  employed  for  the 
application  of  techniques  to  produce  descriptive  statistics. 

Data  analysis  consists  of  performing  tests  and  analytic 
techniques  which  translate  the  data  into  meaningful  results 
(findings,  assessments,  and  suggested  improvements) . 


Section  II 
Test  Organization 


5-5.  Test  Directorate 

OT  is  conducted  by  a  structured  test  directorate  typically 
consisting  of  a  test  director,  deputy  test  director,  test 
officer,  test  analyst,  test  directorate  personnel  provided  as 
specified  in  the  OTP,  and  selected  units.  A  typical  test  unit  or 
organization  structure  is  discussed  below,  however,  the 
configuration  varies  depending  on  the  size  of  the  test  and 
whether  the  system  is  full  evaluate  or  abbreviated  evaluate  or 
whether  the  OT  is  another  type  of  user  test. 

5-6.  Test  team  organization 

The  organization  of  the  test  team  is  tailored  to  fit  the  size, 
scope,  duration,  and  importance  of  the  ter  Careful  thought 
must  be  exercised  to  ensure  optimum  staff i  to  accomplish  each 

task.  The  test  team  must  be  adequate  in  Si  ::  to  function  for  the 
duration  of  the  test  and  man  the  number  of  different  shifts 
required  by  the  test  schedule.  For  control  purposes,  the  test 
team  is  broken  down  into  two  basic  divisions  of  test  organization 
personnel  and  test  player  personnel.  The  following  subelements 
comprise  a  test  team; 

a.  Test  Director /Deputy  Test  Director. 

b.  Test  Of ficer /Deputy  Test  Officer/Test  NCOIC. 


) 
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c.  Administrative  Off icer/Secretary  and  Administrative 
Element. 

d.  Protocol/Visiter  Control/Briefing  Officer. 

e.  Test  Analyst  and  Analytic  Element  (including  MANPRINT, 

RAM,  and  Software) . 

e.  Test  Operations  Officer  and  Operations  Element. 

f.  Chief  Controller  and  Control  Element  (including  Threat 
Manager) . 

g.  Chief  Data  Collector  and  Data  Collection  Element 
(including  MANPRINT,  RAM,  and  AV) . 

h.  Chief  Instrumenter  and  Test  Instrumentation  and  Telemetry 
Element. 

■  i.  Data  Manager  and  Data  Management  Element  (including  Iadp 
Manager) . 

j.  Chief  of  Site  Support  and  Test  Support  Element. 

k.  Troop  Commander  and  Test  Player  Troops. 

l.  Troop  Commander  and  OPFOR  Player  Troops. 

m.  Deputy  Test  Directors  for  Doctrine,  Training,  and 
Logistics  (normally  provided  by  TRADOC  or  other  outside 
activity) . 

n.  MATDEV  Site  Director  and  MATDEV  Test  Support. Team 
(includes  contractor  maintenance  personnel  where  appropriate) . 

o.  Chief  Evaluator  and  Evaluator  Task  Force. 

p.  Data  Authentication  Group  (DAG) 

5-7.  Tailoring  the  test  team 

The  test  team  outlined  above  represents  a  very  large  test.  On  a 
small  test,  many  of  the  jobs  can  be  combined.  The  test  director 
may  only  provide  general  supervision  and  the  test  officer 
provides  all  day-to-day  direction  of  the  test.  The  test  NCOIC 
may  take  charge  of  administration  and  protocol.  The  deputy  test 
officer,  operations  officer,  and  chief  controller  may  be  the  same 
person.  Each  team  should  be  planned  based  on  need  and  not  based 
on  a  prescribed  structure. 
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5-8.  Test  Officer 

a.  For  larger  tests,  the  test  officer  may  be  assisted  by 
assistant  test  officers  for  specific  areas.  Each  assistant  is 
responsible  to  his  organization  for  the  validity  of  the 
appropriate  aspects  of  the  test.  Actual  day-to-day  test 
operations  are  conducted  under  the  supervision  and  management  of 
the  test  officer.  The  test  officer  is  responsible  for  validating 
test  procedures  and  for  preparing  the  test  report.  (See  chapter 
3  of  the  TEP.) 

b.  The  progress  and  ultimate  success  of  a  test  depend  upon 
prior  planning,  support  provided,  proficiency  of  participants, 
and  the  knowledge  and  leadership  of  the  test  officer.  The  job  of 
the  test  officer  is  a  difficult  one  as  no  single  formula  applies 
to  all  tests. 

c.  Tests  roust  be  flexible  and  tailored  to  support  the 
evaluation  of  the  system  or  concept.  In  theory,  if  all  the 
elements  of  the  test  planning  phase  are  feasible  and  properly 
documented  in  the  TEP  and  the  DTP,  test  execution  should  be  a 
smooth  process.  In  practice,  however,  unforeseen  technical 
and/or  administrative  difficulties  nearly  always  occur. 

d.  While  making  every  possible  effort  to  follow  the  test 
design  procedures,  the  test  officer  must  also  be  flexible  enough 
to  adjust  the  program  so  it  can  deal  with  unforeseen  problems  and 
test  anomalies,  such  as  periods  of  lowered  player  motivation, 
instrumentation  failure,  loss  of  key  personnel  on  emergency 
leave,  or  absence  of  key  repair  parts  for  the  tested  prototype. 
This  approach  will  optimize  the  chances  of  success.  It  is 
important  for  the  test  officer  to  ensure  that  the  principal 
agencies  affected  by  any  change  (proponent  and  support 
activities)  are  properly  informed  and  concur  with  the  proposed 
change. 


Section  III 

Detailed  Test  Plan  (Dii ) 


5-9.  General 

The  purpose  of  the  DTP  is  to  supplement  the  TEP,  as  required. 

The  DTP  provides  detailed  instructions,  missions,  tasks, 
organization,  and  procedures  necessary  for  test  execution,  data 
reduction  and  analysis,  and  preparation  of  the  test  report  and 
associated  materials.  It  serves  as  an  informal  "living"  document 
that  is  changed  and  updated  as  required.  It  does  not  require 
publication  or  formal  approval  unless  directed  in  the  TEP. 
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a.  There  are  two  primary  sources  of  information  for  detailed 
test  planning:  the  TEP  and  the  OTP.  The  DTP  is  an  extension  of 
the  TEP  process.  DTP  efforts  may  feed  back  into  the  TEP.  This 
iterative  process  continues  through  the  various  drafts  until  the 
DTP  is  developed. 

b.  Development  of  the  support  plans  appendices  to  the  DTP 
should  move  consistently  toward  more  detail,  in  parallel  with 
refinement  of  the  TEP  and  OTP,  and  should  not  wait  on  revisions 
to  be  published.  These  plans  become  a  part  of  the  DTP. 

c.  The  TEP  contains  information  for  detailed  planning  by 
discussion  of  anticipated  requirements  and  documented  concepts. 
Chapter  3  of  the  TEP  also  contains  some  of  the  most  important 
information  upon  which  to  build  further  planning, 

d.  The  OTP  contains  material  on  all  DTP  supporting  plans. 
Section  1.0,  Part  7  of  the  OTP  defines  test  resource  requirements 
for  instrumentation  support.  "Direct  Cost  Estimates"  section  of 
the  OTP  contains  information  on  AV  support  under  Part  6  -  "Test 
Facilities/Installation,"  and  under  Part  7  -  "Other  Resources 
Required."  Section  1,  Part  4  of  the  OTP  covers  test  resource 
requirements  for  data  collection  and  ADP  support.  Any 
requirements  for  equipment,  personnel,  facilities,  or  ADP  must  be 
included  in  the  OTP  in  order  to  insure  proper  resourcing. 

5-10.  Preparation 

a.  The  format  for  the  DTP  is  shown  in  figure  5-1.  The  DTP  is 
to  include  only  that  material  changed  or  developed  for  the  test 
since  the  preparation  of  the  approved  TEP.  Exceptions  are  for 
paragraphs  1  through  6  of  the  DTP  format  which  repeat  material 
contained  in  chapter  1  of  the  TEP.  This  material  is  necessary  to 
establish  for  the  reader/user  basic  requirements  for  the  test  and 
general  descriptions  of  the  system  to  be  tested. 

(INSERT  FIGURE  5-1) 

b.  Paragraphs  and  appendixes  of  the  approved  TEP  which  have 
been  modified  or  further  developed  as  well  as  planning  material 
developed  subsequent  to  approval  of  the  TEP  should  be  included  in 
the  DTP.  Any  change  to  the  approved  TEP  supported  by  a  formal 
TCP  in  accordance  with  chapter  3  should  be  included  and  the 
approved  TCP  included  in  appendix  S  to  the  DTP.  Changes  to 
chapters  1  through  3  of  the  TEP  should  normally  be  supported  by  a 
TCP  if  the  change  addresses  issues  and  criteria  or  OTMOP,  overall 
OTE  methodology,  required  test  events  or  sample  sizes,  specified 
statistical  confidence  levels,  or  other  major  test  design, 
execution,  or  evaluation  requirements. 
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c.  Paragraph  7  of  the  DTP  provides  the  method  for 
documentation  of  changes  to  chapters  1  through  3  of  the  TEP. 
Changes  can  be  documented  as  subparagraphs  of  paragraph  7  (as 
shown  in  the  example)  or  as  an  appendix  with  entries  similar  as 
those  shown  for  paragraph  7  or  as  copies  of  changed  pages  with 
the  changed  portions  underlined  or  shown  in  bold  type. 

d.  Paragraph  8  and  the  appendix  list  for  the  DTP  provide  the 
method  for  documentation  of  changes  to  the  appendixes  of  the  TEP. 
Brief  descriptions  of  the  extent  of  change  or  development  to  each 
appendix  should  be  contained  in  paragraph  8.  A  complete  copy  of 
the  changed  appendix  with  the  same  appendix  letter  as  for  the 
approved  TEP  will  be  attached  to  the  DTP.  Those  appendixes  not 
changed  from  the  TEP  will  be  identified  as  not  changed  as 
indicated  in  the  format  example.  Additional  appendixes  will  be 
lettered  as  appropriate  following  the  last  letter  used. 

e.  The  intent  of  the  format  just  described  is  to  simplify  the 
development  of  the  DTP  by  eliminating  unnecessary  redundancy. 
However,  the  tester  should  be  aware  that  test  planning  is 
evolutionary  in  nature.  The  TEP  may  embody  the  concept  of  the 
test  but,  for  a  variety  of  reasons,  that  concept  may  not  fully 
mature  until  shortly  before  the  test  starts.  And,  as  the  DTP 
must  serve  as  the  very  blueprint  of  how  that  test  is  to  be 
executed,  the  level  of  specificity  expected  of  it  often  exceeds 
that  of  the  TEP.  Therefore,  a  rigorous  review  of  the  TEP  must  be 
conducted  in  all  planning  areas  to  ensure  that  needed  guidance  is 
available.  Should  that  review  indicate  a  shortcoming,  further 
development  of  requirements  should  be  performed  and  included  in 
the  DTP  as  described. 

5-11.  Review  Process  for  the  DTP 

a.  Internal  review.  Upon  completion  of  the  DTP,  it  should  be 
reviewed,  coordinated,  and  usually  briefed  within  the  test 
iirectorate  responsible  for  planning  and  conducting  the  test. 

ince  practices  vary  among  test  directorates,  the  test  officer 
lould  determine  well  in  advance  the  procedures  he  must  follow 
ior  internal  review. 

b.  External  review.  External  coordination  or  review  of  the 
DTP  is  not  required  unless  specified  by  the  TEP  approval 
document;  however,  it  is  recommended  that  this  document  be 
coordinated  with  the  test  proponent,  operational  evaluator,  and 
TIWG  members  if  time  permits. 

5-12.  Distribution 
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The  test  activity  will  retain  two  copies  of  the  DTP  for  record 
purposes.  Sufficient  copies  will  be  available  in  the  test 
directorate  for  use  by  test  personnel. 

5-13.  Appendixes  to  the  DTP 

The  appendixes  provide  required  copies  of  pertinent  supporting 
documentation,  provide  administrative  listings,  and  expand  TEP 
paragraphs  and  conceptual  appendices  into  detailed  plans  for 
execution. 

5-14.  Content  of  the  appendixes 

a.  Supporting  Documentation.  This  provides  a  listing  of 
pertinent  system  documentation  to  include  the  MNS,  ORD,  TEMP, 

OTP,  the  test  support  packages,  the  TEA  and  the  EIS.  Each 
document  will  be  listed  by  title,  responsible  activity,  status 
(e.g.  approved,  draft,  under  revision) ,  and  dates  of  approval  or 
date  of  draft.  Do  not  attach  copies  of  these  documents  to  the 
DTP. 


b.  Background.  This  optional  appendix  is  included  only  if  it 
appears  in  the  TEP. 

c.  System  Description.  This  optional  appendix  is  included 
only  if  it  appears  in  the  TEP. 

d.  Projected  threat.  This  optional  appendix  is  included  only 
if  it  appears  in  the  TEP. 

e.  DAG  Charter  and  SOP.  A  copy  of  the  DAG  Charter  and  SOP  is 
included  by  the  evaluator  if  requirements  for  a  DAG  are  specified 
in  chapter  2  of  the  TEP.  Details  for  the  preparation  of  a  DAG 
charter  and  SOP  are  contained  in  Section  IV.  For  abbreviated 
evaluations  and  for  tests  of  combat /training  developments,  the 
tester  may  charter  a  DAG  and  add  this  appendix. 

When  the  need  for  a  DAG  is  identified  by  either  the  tester  or  the 
evaluator,  its  mission  and  responsibilities  are  defined  in  this 
appendix. 

f.  OTP.  A  copy  of  the  most  recent  and  approved  OTP 
referenced  in  appendix  A  is  included  as  appendix  F  by  the  tester. 
Where  appropriate,  a  test  resume  sheet  (TRS)  may  be  substituted 
for  the  OTP. 

g.  POA.  The  POA  should  not  change  from  the  TEP. 

h.  Test  scenario(s).  This  optional  appendix  is  included  only 
if  it  appears  in  the  TEP. 
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i.  Test  threat.  This  optional  appendix  is  included  only  if 
it  appears  in  the  TEP. 

j.  Control  Plan.  See  Section  V. 

k.  Data  Management  Plan.  See  Section  VI. 

l.  ITTS  Sypport  Plan.  See  Section  VII. 

m.  Audiovisual  Support  Plan.  See  Section  VIII. 

n.  Automation  Support  Plan.  See  Section  IX. 

o.  Training  Plan.  See  Section  X. 

p.  Test  Support  Plan.  See  Section  XI. 

q.  Test  Environmental  Assessment  (TEA) .  A  copy  of  the 
approved  TEA  is  included  by  the  tester. 

r.  Environmental  Impact  Statement  (EIS) .  A  copy  of  the 
approved  EIS  is  included  by  the  tester. 

s.  DOTE  Approval  of  the  Test  Concept.  Carried  forward  from 
the  TEP. 

t.  Test  Change  Proposals  (TCP) .  Carried  forward  from  the 
TEP.  (See  Chapter  3.) 

u.  Glossary,  Acronyms,  and  Abbreviations.  See  Chapter  3. 

V.  TEP  Coordination  Record.  See  Chapter  3. 

w.  Authors  and  Supporting  Personnel.  See  Chapter  3. 

X.  Distribution.  See  Chapter  3. 


Section  IV 

DAG  Charter  and  SOP 


5-15.  Establishment  of  the  DAG 

The  lOE  establishes  the  requirements  for  a  DAG  on  full-evaluate 
system  tests.  These  requirements  are  documented  in  the  TEP.  If 
the  lOE  does  not  require  a  DAG,  the  tester  determine  if  a  need 
exists  and  establishes  a  DAG.  The  tester  determines  if  a  need 
exists  and  establishes  a  DAG  for  an  abbreviated  evaluate  systems 
and  for  FDTE,  CEP,  and  CT. 
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5-16.  DAG  Charter  and  SOP 

The  originator  of  the  requirement  for  the  DAG  prepares  the  DAG 
charter  and  SOP  in  Appendix  E  of  the  TEP  (e.g.,  when  the  lOE 
requires  a  DAG  for  full-evaluate  system  tests,  the  lOE  prepares 
Appendix  E) .  The  format  for  the  DAG  SOP  is  provided  in  figure  5- 
2.  The  DAG  SOP  states  the  purpose,  responsibilities,  and  the 
products  of  DAG  for  a  specific  test. 

(INSERT  FIGURE  5-2) 


Section  V 

Test  Control  Planning 


5-17.  Test  control  defined 

The  structure  and  procedures  that  will  be  used  to  ensure  that 
required  test  events  occur  in  situations  that  realistically 
depict  the  tactical  context  of  the  test  in  accordance  with  the 
OMS/MP,  the  CBTDEV  prescribed  test  scenarios,  and  the  approved 
threat  for  test. 

5-18.  Determining  requirements  for  test  control 
The  concept  for  test  control  is  jointly  prepared  by  the  chief 
controller  analyst  and  the  test  officer  and  describes  the 
structure  and  procedures  for  control  of  the  test.  It  permits 
quick  identification  and  provides  for  resolution  of  requirement 
misinterpretation  prior  to  test.  If  identified  requirements 
cannot  be  met,  the  issues  are  raised  to  appropriate  levels  for 
resolution. 

5-19.  Documentation  of  test  control  planning 

This  section  provides  guidance  for  incorporation  of  test  control 
planning  for  OTE.  This  planning  results  in  development  of  the 
Test  Control  paragraph  in  the  TEP,  the  Control  Concept  Appendix 
to  the  TEP,  and  the  Control  Plan  Appendix  to  the  DTP.  This 
section  applies  to  all  OT.  Tests  of  major  systems,  with 
extremely  large  test  matrices  and  numerous  test  events,  may 
require  most  of  the  elements  in  the  plan.  Tests  which  are  less 
complex  may  not  require  all  parts  of  the  plan,  however  the  topics 
cans  be  used  as  planning  guides  to  insure  adequate  consideration 
of  these  areas  is  included  in  test  planning. 

5-20.  Test  Control  Planning  Process 

Planning  must  provide  necessary  Test  Control  to  OTE  conduct. 

This  section  describes  the  procedures,  documents,  and  functions 
that  must  be  developed  to  ensure  that  the  test  can  be  properly 
organized,  controlled,  and  executed  to  generate  the  required  test 
data.  The  section  describes  plans  in  detail  but  allows 
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flexibility  for  the  tester  to  tailor  the  plan  for  his  specific 
test. 

5-21.  TEP  control  support  concept 

This  optional  appendix  to  the  TEP  contains  details  of  planned 
test  control  not  contained  in  the  main  body  of  the  document.  It 
is  expanded  into  the  Control  Plan  in  the  DTP.  The  concept  has  no 
fixed  format,  but  using  the  order  of  presentation  given  for  the 
Control  Plan  provides  complete  coverage  of  the  subject  and 
permits  easy  expansion  from  the  concept  to  the  plan. 

5-22.  Documentation  of  the  Control  Plan 

Planning  for  test  control  is  an  iterative  process  which  begins 
with  general  control  measures  and  evolves  into  a  specific  design 
for  control. 

a.  The  central  document  for  test  control  planning  is  the 
Control  Plan.  It  is  prepared  by  the  test  officer  assisted  by  the 
Chief  Controller. 

b.  Detailed  instructions  for  development  of  an  ASP  are 
provided  in  figure  5-3. 


(INSERT  FIGURE  5-3) 

5-23.  Test  Control  Concept 

This  section  of  the  plan  describes  the  control  concept.  This 
paragraph  includes  descriptions  of  the  control  structure  and 
procedures  that  will  be  used  to  ensure  that  required  test  events 
occur  in  situations  that  realistically  depict  the  tactical 
context  of  the  test  in  accordance  with  the  OMS/MP. 

5-24.  Level  of  Operational  Realism 

The  choice  among  the  three  levels  of  operational  realism  should 
be  stated  along  with  a  short  rr-  onale  for  the  choice.  These 
levels  are  (see  Chapter  3  for  i  jnitions) : 

a.  Maximum  Operational  Realise. 

b.  Limited  Operational  Realism. 

c.  Minimal  Operational  Realism. 

5-25.  Synopsis  of  Events 

An  overview  of  the  proposed  test  events  and  the  environment  and 
sequence  in  which  they  will  occur.  For  maximum  operational 
realism,  a  narrative  of  the  major  tactical  operations  and  their 
sequence  is  valuable  in  assessing  realism.  For  tests  involving 
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limited  operational  realism,  the  tasks  or  missions  to  be 
performed  are  to  be  described. 

5-26.  Control  Methods 

An  overview  of  the  methods  to  be  employed  in  causing  the  test 
events  to  happen. 

5-27.  Program  of  Events 

This  section  provides  both  an  outline  of  what  should  occur  during 
the  test  and  a  detailed  time  schedule  of  explicit  events. 

5-28.  Overall  Test  Scenario 

a.  This  outline  divides  the  test  into  days,  mission  profiles, 
events,  subtests,  phases,  or  any  other  convenient  divisions.  A 
day  or  a  mission  may  be  repeated  a  number  of  times  to  develop 
sufficient  data  as  required  by  the  event  matrices.  Any  such 
repetition  should  be  scheduled  to  agree  with  ordering  constraints 
or  randomization  requirements  in  the  matrix. 

b.  This  outline  reserves  periods  within  which  events  occur 
and  serves  as  a  tool  for  coordination.  General  procedures  are 
developed  to  address  delays  or  other  events  which  preclude 
accumulation  of  scheduled  data  due  to  weather,  instrumentation 
failures,  system  failures,  or  other  such  events. 

5-29.  Detailed  Test  Scenario 

a.  The  detailed  test  scenario  consists  of  a  time  schedule  of 
events,  a  description  of  events,  and  associated  documents  for 
event  inputs.  Based  on  the. overall  test  scenario,  realistic 
operational  events  are  scheduled  into  appropriate  time  blocks 
using  enemy  threat  documentation,  U.S.  doctrine  and  tactics,  and 
test  constraints. 

b.  For  test  scenario  purposes,  an  event  is  something  that  can 
be  controlled  and  scheduled  ^o  occur  at  a  specified  time.  For 
example,  the  initial  conditions  (time,  location,  tactics,  force 
sizes)  for  an  opposing  force  ambush  of  a  friendly  infantry 
platoon  can  be  specified  in  advance,  so  such  an  ambush  can, 
therefore,  be  considered  an  event.  However,  the  resultant 
actions  following  the  ambush  are  generally  unpredictable,  so  they 
would  be  regarded  as  test  outcomes  rather  than  test  events. 

c.  Appropriate  documents  used  to  support  or  initiate  test 
scenario  events  may  be  operations  orders,  intelligence  reports, 
fragmentary  plans,  and  simulated  enemy  documents.  The  completed 
detailed  scenario  is  the  principal  control  mechanism  for  the 
test. 
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d.  A  major  function  of  the  pilot  test  is  to  examine  the 
adequacy  of  this  mechanism  and  identify  changes  necessary  for  the 
actual  test.  Procedures  are  developed  to  monitor  scenario 
execution  during  the  actual  test  to  verify  that  programmed  events 
are  executed  in  accordance  with  the  detailed  scenario.  The 
organization  and  level  of  detail  of  the  program  of  events  depend 
upon  the  level  of  simulated  realism  required. 

5-30.  Full  Tactical  Simulation. 

a.  For  tests  which  require  simulation  of  a  tactical 
environment  for  extended  periods,  the  program  of  events  will 
consist  of  a  complete  scenario.  Essentially,  a  scenario  is  a 
narrative  which  merges  the  test  events  into  a  realistic  and 
believable  sequence.  A  scenario  describes  the  actions  of  sli 
player  and  OPFOR  units  and  includes  all  information  which  will  be 
presented  to  the  players.  This  information  includes  initial  and 
update  situation  briefings,  operations  orders,  fragmentary 
orders,  intelligence  summaries,  messages,  and  other  information 
designed  to  evoke  player  response. 

b.  Particular  attention  must  be  paid  to  the  actions  of  OPFOR. 
Their  operations  should  in  all  cases  be  consistent  with  the 
tactics  of  the  threat  force  being  considered. 

c.  All  scenarios  must  be  based  on  one  of  the  TRADOC  standard 
scenarios.  The  particular  standard  scenario  to  be  used  is  agreed 
upon  by  the  test  organization  and  the  proponent,  usually  during 
the  OTP  preparation  or  the  preliminary  test  planning  phase.  In 
preparing  the  scenario,  it  is  essential  to  specify  the  time  and 
location  of  each  planned  event. 

5-31.  Limited  Tactical  Simulation. 

a.  7  ests  which  do  not  require  maximum  combat  realism,  the 
preparat  of  a  scenario  is  unnecessary.  It  is,  however, 
necessary  to  develop  a  detailed  description  of  the  events  that 
will  occur.  The  description  should  be  sufficiently  detailed  so 
that  the  events  can  be  properly  executed  without  additional 
information. 

b.  For  each  event  the  method,  time,  location,  participants, 
and  information  to  be  provided  must  be  specified.  It  is  often 
possible  to  group  similar  or  near  simultaneous  events  into 
subtests.  This  reduces  the  volume  of  necessary  description  and 
makes  conduct  of  the  test  more  efficient. 

5-32.  No  Tactical  Simulation 
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For  tests  such  as  user  evaluations  and  some  custdmer  tests,  no 
simulated  realism  is  required.  For  such  tests,  the  event  summary 
can  be  augmented  as  necessary  to  provide  a  description  of  the 
conditions  or  situations  planned  or  expected  during  the  test. 
Examples  include  identification  of  test  units,  organization  of 
the  test  into  phases,  and  expected  frequency  of  key  events. 

5-33.  Control  Procedures 

The  program  of  events  describes  in  detail  the  events  which  must 
occur.  The  control  procedures  support  the  program  of  events  by 
specifying  the  system  which  will  cause  the  events  to  occur.  The 
content  of  the  control  procedures  depends  upon  the  level  of 
simulated  realism  required  in  the  test. 

5-34.  Control  Procedures  for  Full  Tactical  simulations 
For  tests  of  full  tactical  simulation,  the  primary  control 
objective  is  to  induce  the  test  players  to  comply  with  the 
scenario.  In  support  of  this  objective,  the  control  procedure 
should  describe  the  control  organization  and  responsibilities, 
the  scenario  revision  and  documentation,  and  the  casualty 
assessment  requirements. 

5-35.  Scenario  Revision  and  Documentation. 

a.  Since  the  scenario  is  a  very  detailed  document,  it  is  to 
be  expected  that  changes  may  become  necessary  during  the  test. 
Major  changes  may  require  proponent  concurrence.  When 
information  from  field  controllers  indicates  that  a  change  has 
occurred  or  will  occur,  the  control  headquarters  should  be 
prepared  to  revise  the  scenario  in  order  to  ensure  the  occurrence 
of  required  events. 

b.  This  section  of  the  control  procedure  should  contain 
guidelines  for  revising  the  scenario  and  should  direct  procedures 
for  documenting  the  revisions.  The  best  procedures  for 
documenting  a  revision  is  to  keep  a  log  within  the  control 
headquarters.  This  log  should  list  all  test  events,  by  time  and 
location,  as  well  as  directed  revisions.  This  log  then  becomes  a 
revised  scenario. 

5-36.  Casualty  Assessment. 

An  essential  element  of  full  tactical  simulation  is  the  method 
used  for  assessing  casualties  during  engagements.  When 
instrumentation  systems  are  used,  this  section  should  describe 
the  vehicle  and  weapon  simulations.  Guidance  on  employment  of 
instrumentation  systems  must  be  obtained  from  the  agency  or 
office  that  maintains  and  operates  that  system.  If  controllers 
assess  casualties,  then  the  procedure  roust  be  specified  in 
detail.  Guidance  on  assessment  procedures  can  be  found  in  plans 
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of  prior  tests  and  in  FM  105-5.  The  assessment  of  casualties  is 
of  crucial  importance,  not  only  because  it  directly  affects  test 
data  but  also  because  it  has  a  major  effect  on  troop  enthusiasm 
and  morale. 

5-37.  Control  Procedures  for  Limited  Tactical  Simulations 
The  control  procedures  for  limited  simulation  tests  is  similar  to 
that  required  in  cases  of  full  tactical  simulation.  It  is 
usually,  however,  less  elaborate  since  the  control  function  is 
less  complicated.  Since  this  category  covers  a  wide  range  of 
tests,  the  best  guidance  in  planning  the  control  procedures  will 
be  obtained  from  plans  of  similar  prior  tests.  As  a  minimum  the 
document  should  include  the  control  organization  and  functions,  a 
method  for  revising  the  program  of  events,  and  a  system  for 
documenting  the  revisions.  Casualty  assessment  may  be  required 
for  some  tests  of  this  type. 

5-38.  Control  Procedures  With  No  Tactical  Simulation 

For  tests  with  no  simulation,  the  control  procedure  consists,  as 

a  minimum,  of  two  parts. 

a.  The  first  part  describes  the  supervision  and  monitoring 
that  the  test  officer  or  other  controllers  will  perform  to  assure 
that  the  test  item  or  concept  is  being  employed  properly.  The 
description  should  include  a  schedule  of  visits  or  inspections 
and  a  listing  of  points  to  be  checked.  Such  frequent  checks  are 
necessary  to  assure  that  misuse  or  misunderstanding  does  not  bias 
results  of  the  test. 

b.  The  second  part  should  outline  contingency  procedures  to 
be  employed  if  any  required  events  do  not  occur  in  the  course  of 
the  test  unit's  operations.  These  procedures  will  often  involve 
some  limited  simulation  testing.  Any  additional  information 
which  would  enhance  understanding  of  the  conduct  of  the  test 
should  be  included. 

5-39.  Pilot  Test 

a.  A  pilot  test  is  an  abbreviated  version  of  the  actual 
test  and  is  conducted  in  advance  to  detect  deficiencies  in  the 
plan.  For  this  reason  it  should,  as  a  minimum,  include  the 
exercise  of  each  type  of  required  event  and  make  use  of  each  data 
collection,  data  reduction  form,  and  mechanism  to  be  used  in  the 
actual  test. 

b.  The  pilot  test  should  be  provided  for  in  the  control 
plan  with  sufficient  time  between  pilot  test  and  the  start  date 
of  the  actual  test  to  allow  for  reaction  and  correction  of  any 
deficiencies  encountered  in  control,  data  collection,  and  data 
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reduction.  Tests  relying  heavily  on  instrumentation  may  require 
additional  time  after  the  pilot  test  for  the  correction  of 
problems . 

c.  Accomplishment  of  an  abbreviated  program  of  events  is 
usually  sufficient,  although  an  abbreviated  control  procedure  may 
also  be  required. 

5-40.  Organization  for  Control 

The  control  organization  and  responsibilities  section  will 
describe  the  structure  of  the  control  group  down  to  individual 
level.  The  required  background  and  qualifications  for  each 
member  of  the  organization  will  be  specified.  The  mission  and 
responsibilities  of  each  subunit  will  be  described  in  detail. 

a.  No  single  organization  is  applicable  to  all  tests.  Every 
control  organization  requires  a  centralized  headquarters  to  issue 
instructions,  reallocate  assets,  and  monitor  progress. 

b.  In  addition,  field  controllers  are  required  to  provide 
information  to  the  headquarters  and  to  intervene,  if  necessary. 
These  field  controllers  may  accompany  the  units  or  they  may  be 
assigned  to  locations  in  the  test  area.  Occasionally,  a 
combination  of  unit  controllers  and  area  controllers  is 

^  necessary. 

c.  The  test  officer  must  develop  the  control  organization 
plans.  This  plan  describes  the  organization  of  the  control 
section,  including  functions  and  responsibilities  of  key  section 
personnel  and  plans  for  the  briefing,  debriefing,  replacement, 
relief,  and  rotation  of  controllers.  It  also  describes  the 
formal  control  documentation  required,  including  test  status 
reports,  historical  records,  and  submission  schedules. 

d.  Controllers  maintain  high  test  quality  by  ensuring  that 
required  test  conditions  are  met.  If  the  test  is  a  product 
improvement  to  an  old  system,  use  of  controllers  from  a  unit 
similar  to  the  player  unit  is  encouraged.  For  a  new  system, 
controllers  may  come  from  a  school,  test  directorate,  or  other 
unit. 


e.  The  control  section  usually  consists  of  teams  with  one 
team  assigned  to  each  kind  of  system  or  major  end  item.  Each 
team  requires  enough  personnel  to  allow  for  division  and  rotation 
of  duties.  Since  opposing  forces  are  usually  controlled  by  the 
scenario,  they  are  typically  placed  under  jurisdiction  of  the 
control  section. 


( 


5-15 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992  (DRAFT) 


f.  The  controllers  are  responsible  for  recommending  a  halt 
in  testing  when  incidents  or  problems  emerge  which  significantly 
limit  achievement  of  test  objectives  or  when  a  serious  safety  or 
health  hazard  is  identified. 

g.  Controllers  maintain  a  log  of  events  or  scenarios  which 
record  the  conditions  which  existed  during  the  test  as  opposed  to 
what  was  planned.  This  log  is  separate  from  the  task  of  the  data 
collectors  who  record  what  resulted  from  the  test  conditions. 

5-41.  Communications  Control  •  ^  a. 

This  section  describes  the  communications  systems  required  to 
control  and  support  the  test.  The  core  of  the  communications 
plan  is  the  development  of  radio  and  wire  nets  for  test  control, 
data  collection,  and  player  communication  purposes.  Frequency 
allocation  is  to  be  done  early  and  in  close  coordination  with 
appropriate  local  authorities  to  preclude  unanticipated 
interference  with  military  or  civilian  communications  or  test 
instrumentation.  It  is  always  better  to  have  more  frequencies 
than  needed,  rather  then  risk  comprise  in  control  or  safety  for 
lack  of  alternatives. 

5-42.  Operations  Center 

This  plan  describes  standing  operating  procedures  for  the  test 
operations  center,  including  hours  of  operation  and  staffing 
levels.  The  test  operations  center  facilitates  minute-by-minute 
control  of  testing  and  requires  the  capability  to  obtain, 
coordinate,  and  communicate  authoritative  decisions  quickly  for 
"hold-scrub-substitute”  questions,  for  test  incidents,  or^ 
emergencies.  The  test  operations  center  maintains  a  detailed  log 
of  all  test  events  and  control  decisions. 


Section  VI 

Data  Management  Planning 


5-43.  Data  Management  Defined 

Data  management  is  the  collection,  quality  control,  reduction, 
and  storage  of  test  data  generated  during  test  execution  into  a 
more  condensed  and  usable  format  for  analysis  and,  ultimately, 
into  test  results  and  evaluation. 

5-44.  Determining  Requirements  for  Data  Management 
The  concept  for  data  management  is  jointly  prepared  by  the  chief 
data  collector,  the  data  manager,  the  test  analyst,  and  the  test 
officer  and  describes  support  to  be  provided  in  order  to  meet  the 
data  requirements  of  the  test.  It  permits  quick  identification 
and  resolution  of  requirement  misinterpretation  prior  to  test. 
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If  identified  requirements  cannot  be  met,  the  issues  are  raised 
to  appropriate  levels  for  resolution. 

5-45.  Documentation  of  Data  Management  Planning 

This  section  provides  guidance  for  incorporation  of  data 

management  planning  for  OTE.  This  planning  results  in 

management  section  in  the  TEP,  the  Data 

the  Data  Management 

n  Appendix  to  the  DTP.  This  section  applies  to  all  OT.  Tests 

extremely  large  data  bases  and  highly 
th2^n??2”^®2  collection,  may  require  most  of  the  elements  in 

of  which  are  less  complex  may  not  require  all  parts 

Tl!  the  topics  cans  be  used  as  planning  guides 

test  planning.  *  areas  is  included  in 

5-46.  Data  Management  Planning  Process 

provide  necessary  details  of  OTE  data  management. 
d^tJiiJ  of  H  support  planning  occurs,  and  discusses 

The  iectifn  write  plans  for  data  management. 

fhJ  ®®^t ion  describes  plans  in  detail  but  allows  flexibility  for 
the  tester  to  tailor  the  plan  for  his  specific  test. 

5-47 •  TEP  Data  Management  Concept 

appendix  to  the  TEP  contains  details  on  planned 
data  management  not  contained  in  the  main  body  of  the  document 
It  IS  expanded  into  the  Data  Management  Plan  in  the  DTP  The 
concept  has  no  fixed  format,  but  using  the  order  of  presentation 
g  ven  for  the  plan  provides  complete  coverage  of  the  subiect  and 
permits  easy  expansion  from  the  concept  to  the  plan. 

5-48.  Documentation  of  the  Data  Management  Plan 
Planning  for  data  management  is  an  iterative  process  which  begins 
with  general  data  requirements  and  evolves  into  a  specific  design 
for  data  collection,  reduction,  storage,  and  qualiS  control  ^ 

J^®  insure  a  concept  which  satisfies 

data  requirements  and  provides  bases  of  understanding  between 
sters,  evaluators,  and  the  data  manager  concerning  data 

instructions  for  development  of  a  Data 
Management  Plan  are  provided  in  figure  5—4. 

(INSERT  FIGURE  5-4) 

5-49.  Data  Management  Concept 

describes  the  data  management  concept 
and  summarizes  the  system.  This  first  step  provides  the  basis 
for  the  initial  part  of  the  plan.  This  conLprsiiould  b2  2?Kten 
in  layman  s  terms,  and  should  describe  the  system  with  respect  to 
organization,  flow  of  data,  and  methods  of  operation  that  the 
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data  collectors,  data  reducers,  and  data  entry  personnel  are 
required  to  follow.  Concept  development  begins  the  iterative 
process  for  planning,  which  is  refined  and  expanded  as  the  TEP  is 
further  refined.  The  concept  is  composed  of  the  following 
subsections: 

a.  Organization  for  Data  Management. 

b.  Data  Flow  and  Anticipated  Volume  of  Data 

c.  Methods  of  Operation. 

5-50.  Data  Collection  Procedures 

This  section  describes  the  system  and  the  procedures  the  test 
team  will  use  to  meet  the  requirements  for  data  management. 

a.  Collection  Methods.  Each  method  of  data  collection 
intended  for  the  test  is  identified.  Under  each  method  is  listed 
the  general  categories  of  data  it  will  be  used  to  collect. 

Methods  to  be  considered  include  manual  recording  by  players, 
manual  recording  by  data  collectors,  instrumentation, 
questionnaires,  and  structured  interviews. 

b.  Collection  support.  Any  special  requirements  pertaining 
to  data  collection  are  identified.  Examples  include  the 
following  items: 

(1)  Overview  of  test  organization  and  planned  initial  quality 
control  procedures.  This  includes  daily  debriefing  of  manual 
data  collectors  and  quality  control  checks  of  data  collection 
forms  to  ensure  completeness. 

(2)  Instrumentation  requirements,  to  include  software 
development  if  required  (in  coordination  with  the  Instrumentation 
Support  Plan) . 

(3)  Data  collec, ors  with  special  qualifications. 

(4)  Expected  security  and  special  data  handling  requirements. 

(5)  Video  and  photography  documentation  (in  coordination  with 
the  Audiovisual  Support  Plan) . 

5-51.  Data  Reduction  Procedures 

The  procedures  through  which  recorded  test  data  is  organized, 
reduced,  verified,  managed,  controlled,  and  stored.  Each  data 
requirement  and  its  means  of  collection  are  delineated  in  the 
body  of  the  TEP.  The  tester  organizes  the  data  in  terms  of  the 
collection  source  (RAM  collection  form,  cockpit  digital  recorder, 
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radar  tapes,  etc.).  The  result  of  the  data  reduction  process  is 
level  3  data. 

5-52.  Data  Entry 

There  are  two  basic  formats  for  entering  data  into  a  data  base: 
manual,  and  automatic  feed.  The  extent  to  which  each  of  these 
methods  will  be  used  in  the  data  reduction  of  the  test  should  be 
expressed  in  the  plan. 

a.  Manual.  Manual  entry  occurs  when  data  is  generated  from 
forms  and  questionnaires,  usually  measuring  RAM  data  or  human 
factors  data. ^  Forms  and  questionnaires  should  be  standardized 
and  made  as  simple  and  objective  as  possible  in  order  to 
facilitate  the  entry  of  the  data  into  the  ADP  system. 

Furthermore,  some  type  of  quality  control  of  the  manual  data 
should  be  planned  to  insure  that  only  accurate  data  is  entered 
into  the  data  base,  and  that  no  typographical  errors  occur.  This 
section  should  describe  the  data  that  will  be  entered  by  hand, 
exactly  how  it  will  be  entered,  which  personnel  will  check  it, 
and  which  personnel  will  enter  it. 

b.  Automated.  Data  is  entered  by  automation  when  it  has  been 
generated  and  collected  by  instruments  which  are  tied  directly 
into  the  ADP  system,  or  collected  and  stored  on  some  ADP  medium. 
Here,  a  quality  control  plan  should  be  devised  to  check  the 
linkage  between  the  instrumentation  and  the  ADP  system.  For  this 
type  of  entry,  the  plan  should  describe  the  data,  the  path  it 
will  take,  where  and  how  it  will  be  controlled,  and  which 
personnel  will  control  it, 

5-53.  Data  Processing 

This  next  step  in  the  processing  of  data  may  consist  of  loading 
data  into  storage  or  into  an  organized,  readable  form. 

Procedures  for  this  task  should  be  included,  if  applicable. 

5-54.  Data  Verification/Validation 

The  plan  should  specify  the  procedures  for  verifying  and 
validating  the  data  being  loaded  into  the  data  base.  This 
includes  answers  to  where,  when,  how  and  by  whom  these  tasks 
would  be  accomplished. 

5-55.  Data  Storage 

While  data  is  being  validated  by  the  data  manager,  it  will  be 
stored  in  special  temporary  files  which  cannot  be  processed  by 
the  users.  After  validation,  the  data  will  be  uploaded  into  the 
data  base  of  the  host  computer  for  permanent  storage. 

5-56.  Data  Manipulation 
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In  some  cases  the  form  of  the  data  may  need  to  be  altered  before 
it  can  be  shipped  to  another  data  base,  or  be  summarized  and 
analyzed.  These  considerations,  along  with  the  details  for 
accomplishing  this  manipulation,  should  be  specified. 

5-57.  Data  Presentation 

Identify  the  concept  for  data  display  requirements  including  both 
management  displays  and  test  report  displays.  Management 
displays  summarize  the  status  of  the  test  or  test  events.  Test 
report  displays  present  or,  in  special  cases  summarize,  test 
data. 

5-58.  Data  Base  Design 

Describe  the  concept  for  structure  and  content  of  the  automated 
data  base.  If  available  at  this  time,  the  description  of  the 
test  data  base  will  include: 

.a.  Identification  of  the  required  files  to  include  labels,  a 
short  description  of  the  type  of  data  to  be  stored  in  each  file, 
and  an  indication  for  each  file  whether  it  is  to  be  automated. 

b.  A  description  of  the  architecture  used  to  organize  the 
data  base.  The  architecture  typically  consists  of  either  a 
network  or  relational  design  for  data  storage. 

(1)  The  network  approach  requires  specification  of  both  the 
types  of  data  as  well  as  the  one-to-many  or  one-to-one 
relationships  between  data.  Such  designs  may  not  be  appropriate 
for  operational  evaluation  applications  because  only  data 
previously  related,  by  explicitly  defined  associations,  can  be 
processed. 

(2)  The  relational  approach  organizes  data  into  arrays  of 
rows  and  columns.  Only  the  types  of  data  need  to  be  specified; 
the  potential  relationships  between  data  are  implicitly 
represented.  This  approach  permits  the  rapid  generation  of  a  new 
file,  by  combining  across  two  or  more  types  of  data. 

c.  Definition  of  data  elements  in  each  file  to  include 
sufficient  definitions  to  preclude  misunderstanding  and  ambiguity 
(Data  Element  Dictionary) . 

5-59.  Data  Inputs  and  Outputs 

This  is  a  description  of  the  data  inputs  and  outputs.  In  a  data 
base  of  any  size  this  subparagraph  should  give  a  basic 
description  and  refer  the  reader  to  an  appendix  where  each 
element  is  listed.  It  is  related  to  and  may  be  cross-referenced 
to  the  Data  Base  Dictionary.  It  should  include  the  following 
data. 
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a.  Data  name:  A  generic  or  external  name  for  the  data 
element. 

b.  System  name:  The  proper  name  used  in  the  data  base.  This 
is  the  one  that  must  be  used  when  working  with  the  data  base. 

c.  Width  (char) :  The  minimum  and  maximum  length  in 
characters  when  the  item  is  printed. 

d.  Format:  The  type  of  data  in  the  element.  A=alphabetic, 
N=numeric,  AN=alphanumeric,  Y=year. 

e.  Edit  checks:  Specifies  the  edit  checks  the  system  will 
perform. 

f.  Source:  The  original  source  of  the  data. 

5-60.  Data  Base  Structure 

This  section  of  the  plan  should  finalize  the  organization  and 
structure  of  the  test  data  base.  It  should  also  detail  the 
following  considerations: 

a.  Dictionary.  The  structure  of  the  dictionary  may  vary  from 
test  to  test  but  should  generally  contain  the  data  name,  the 
system  name,  the  data  description,  the  range  of  the  data,  and  the 
data  source. 

b.  Sample  Forms.  The  data  base  should  be  structured  using 
sample  collection  forms  to  record  RAM  factors  as  well  as  other 
factors  as  they  develop. 

5-61.  Data  Base  Management 

This  paragraph  delineates  the  procedures  for  accessing  and 
changing  the  data  base.  The  activities  or  individuals  who  can 
query,  manipulate,  view,  or  modify  all  or  part  of  the  data  base 
or  other  test  results  will  be  delineated.  The  authority  to 
release  all  or  part  of  the  data  will  also  be  specified. 

5-62.  Quality  Control 

This  paragraph  outlines  the  procedures  for  independently 
validating  test  data.  It  defines  the  checks  and  procedures  which 
are  to  be  used  by  the  tester  to  preclude  or  detect  and  correct 
errors  made  in  data  collection,  data  entry,  or  data  reduction. 

a.  Both  manual  and  automated  checks  are  used  to  ensure  that 
data  are  correctly  entered  and  test  events  are  accurately 
portrayed.  Although  manual  checks  often  include  extensive  review 
of  computer  printouts,  they  should  not  generally  involve 
extensive  arithmetic  operations.  Such  operations,  together  with 
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repetitive  logical  checks,  should  be  automated  to  the  maximum 
extent.  Manual  checks  consist  primarily  of  checks  which  cannot 
be  easily  automated  (such  as  review  of  graphical  displays) . 

b.  This  paragraph  also  identifies  emerging  data  summaries 
required  to  identify  potential  inconsistencies.  It  outlines  the 
concept  for  making  required  corrections  in  test  data  and  how  an 
audit  trail  for  those  corrections  is  to  be  maintained. 

c.  Data  validation  determines  whether  the  data  correctly 
represent  the  variables  they  characterize  and  whether  the  data 
collected  are  adequate.  This  paragraph  specifies  requirements 
for  and  techniques  proposed  for  data  validation.  Data  collection 
procedures  are  to  be  validated  prior  to  starting  the  test  and 
checked  during  the  test  to  ensure  that  critical  data  are  being 
accurately  collected. 

5-63.  Output  Reports 

The  plan  should  detail  how  the  data  will  be  reported.  Some 
report  considerations  are: 

a.  Test  Report.  This  report  is  the  main  document  stating  the 
processed  results  of  the  test. 

b.  Data  Displays.  The  plan  should  specify  how  the  data  will 
be  displayed,  in  accordance  with  the  TEP.  Data  displays  should 
be  identified  and  demonstrated  as  early  as  possible  in  the 
process  to  insure  capability  to  present  data  in  conjunction  with 
the  evaluator's  planned  TER.. 

c.  Other  Preliminary  Reports.  The  plan  should  discuss 
provisions  for  other  preliminary  reports  of  testing  results  such 
as  DAG  reports.  Quality  Control  reports.  Scoring  Conference 
reports,  and  Test  Incident  Reports. 


Section  VII 

ITTS  Support  Planning 


5-64.  ITTS  Support  Defined 

ITTS  are  scientific  or  technical  equipment,  to  include  threat 
simulators  and  targets,  used  to  measure,  sense,  record,  transmit, 
process,  or  display  data  during  tests,  examination  of  materiel, 
training  concepts  or  tactical  doctrine. 

a.  Instrumentation.  Electro-magnetic  (e.g.,  electrical, 
electronic,  laser,  radar,  photosensitive,  etc.)  and  other 
equipment  (e.g.,  optical,  electro-optical,  audio,  mechanical. 
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automated  information,  etc.)  used  to  detect,  measure,  record, 
telemeter,  process,  or  analyze  physical  parameters  or  quantities 
encountered  in  the  T&E  process.  Instrumentation  may  apply  to  a 
system  under  test  or  to  a  target  or  threat  simulator. 

(1)  Major  Instrumentation.  That  instrumentation  which 
satisfies  joint  service  requirements,  serves  multiple  Army 
commands,  requires  a  significant  level  of  development  and 
integration,  or  has  a  large  dollar  value.  Major  Army 
instrumentation'  acquisition  is  normally  PM  managed. 

(2)  Sustaining  Instrumentation.  That  instrumentation  which 
is  not  defined  as  major  and  which  satisfies,  within  a  single 
command,  routine  or  recurring  needs  and  normally  has  a  lower 
acquisition  cost  than  major  instrumentation.  Sustaining 
instrumentation  is  normally  acquired  by  the  requiring  command. 

b.  Targets.  Targets  are  normally  economical,  expendable 
devices  used  for  tracking  and  engagement  by  missiles  or  munitions 
in  support  of  T&E  as  well  as  training  missions.  Drone  targets 
are  air  or  ground  vehicles  converted  to  remote  or  programmable 
control.  Ground  targets  are  intended  to  represent  an  adversary 
ground  vehicle  system  or  ground  based  military  structure.  Aerial 
targets  are  intended  to  represent  adversary  aircraft.  Targets 

'  may  represent  only  selected  adversary  system  characteristics. 

c.  Threat  Simulator.  Threat  simulator  is  a  generic  term  used 
to  describe  equipment  which  represents  adversary  systems.  A 
threat  simulator  has  one  or  more  characteristics  which,  when 
detected  by  human  senses  or  man-made  sensors,  provide  the 
appearance  of  an  actual  adversary  system  with  a  prescribed  degree 
of  fidelity.  Threat  simulators  are  not  normally  expendable. 

5-65.  Need  for  ITTS  Support 

Each  OTE  iteration  requires  an  ITTS  Support  Plan  (ISP)  which 
results  in  adequate,  effective,  and  timely  ITTS  support  for 
testing.  The  collection  of  test  data  through  the  use  of 
automated  instrumentation  systems  is  a  key  factor  in  the  majority 
of  tests  conducted.  In  addition,  instrumentation  systems,  to 
include  models  and  simulations,  are  often  employed  to  provide 
realistic  simulation  of  combat  environments  (weapons  simulator, 
NBC  simulants,  etc.)  and  to  generate  data  for  systems  to  use  in 
lieu  of  having  actual  forces  in  the  field  (combat  simulations  or 
stimulation  models) . 

5-66.  Determining  Requirements  for  ITTS  Support 
The  concept  for  ITTS  support  is  jointly  prepared  by  the  threat 
analyst,  the  instrumentation  engineer  and  the  test  officer  and 
describes  the  support  to  be  provided  in  order  to  meet  the  ITTS 
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requirements  of  the  test.  It  permits  quick  identification  and 
resolution  of  requirement  misinterpretation  prior  to  test.  If 
identified  requirements  cannot  be  met,  the  issues  are  raised  to 
appropriate  levels  for  resolution. 

5-67.  Documentation  of  ITTS  Support  Planning 

This  section  provides  guidance  for  incorporation  of  ITTS  support 
planning  for  OTE.  This  planning  results  in  development  of  the 
ITTS  support  paragraphs  in  the  TEP,  the  ITTS  Support  Concept 
Appendix  to  the  TEP,  and  the  ITTS  Support  Plan  Appendix  to  the 
DTP.  This  section  applies  to  all  OT.  Tests  of  major  systems, 
with  extremely  large  data  bases  and  highly  instrumented  data 
collection,  may  require  most  of  the  elements  in  the  plan.  Tests 
which  are  less  complex  may  not  require  all  parts  of  the  plan, 
however  the  topics  cans  be  used  as  planning  guides  to  insure 
adequate  consideration  of  these  areas  is  included  in  test 
planning. 

5-68.  ITTS  Support  Planning  Process 

Planning  must  provide  necessary  ITTS  support  to  OTE  data 
management  requirements.  This  section  explains  how  support 
planning  occurs,  and  discusses  details  of  how  to  develop  and 
write  support  plans  for  ITTS  support.  The  section  describes 
plans  in  detail  but  allows  flexibility  for  the  tester  to  tailor 
the  plan  for  his  specific  test. 

5-69.  TEP  Automation  Support  Concept 

This  optional  appendix  to  the  TEP  contains  details  of  planned 
ITTS  support  not  contained  in  the  main  body  of  the  document.  It 
is  expanded  into  the  ISP  in  the  DTP.  The  concept  has  no  fixed 
format,  but  using  the  order  of  presentation  given  for  the  ISP 
Plan  provides  complete  coverage  of  the  subject  and  permits  easy 
expansion  from  the  concept  to  the  plan. 

5-70.  ITTS  Scheduling 

Development  of  an  implementation  sch  le  is  paramount  to  an  ISP 
plan.  This  schedule  must  be  in  cons^  ace  with  the  overall  test 
schedule  to  allow  adequate  planning  and  development  time  for 
support  of  the  test.  The  support  plan  is  the  responsibility  of 
the  tester  with  input  from  the  evaluator.  The  tester,  in  turn, 
assigns  a  threat  analyst  and  an  instrumentation  engineer  with  the 
requisite  technical  expertise  to  help  prepare  the  plan. 

5-71.  Preparation  of  the  ISP 

a.  When  ITTS  support  is  required  for  test  scenarios  and  data 
collection,  the  requirements  are  prepared  by  the  tester  and 
supporting  threat  and  instrumentation  elements.  They  describe 
the  support  to  be  provided  in  order  to  meet  the  ITTS  requirements 
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of  the  test.  Advance  planning  permits  quick  identification  and 
resolution  of  requirements  misinterpretation  prior  to  the  test. 

b.  The  support  needed  is  based  on  automation  requirements  and 
threat  portrayal  developed  during  overall  test  planning.  ITTS 
planning  is  conducted  to  identify  those  ITTS  systems  that  are 
required  to  collect  data  to  address  the  test  issues  and  provide 
the  necessary  degree  of  combat  environment  realism  or  generation 
of  cue/ task  loading  information. 

c.  The  test  officers  must  identify,  through  the  overall  data 
requirements  identification,  test  control  procedures  and  data 
collection  reduction  planning,  the  detailed  requirements  for  ITTS 
support.  Many  sources  exist  to  assist  the  test  officer  in 
identification  of  available  systems  and  system  capabilities. 

d.  Test  officers  should  coordinate  with  the  instrumenters  as 
early  in  the  test  planning  phase  as  possible  for  identification 
of  requirements.  Instrumentation  engineers  will  assist  the  test 
officer  in  the  development  of  all  instrumentation  requirements 
for  the  test.  Details  of  each  ITTS  system  to  be  used  in  the  test 
must  be  addressed.  The  system  description  includes  information 
on: 

(1)  Instrumentation  source  and  disposition. 

(2)  Number  and  qualifications  of  operators  and  maintainers 
required  for  each  shift. 

(3)  Appropriate  points  of  contact;  schedules  of  critical 
acquisition  or  disposition  events. 

(4)  Test  site  support  requirements  (including  space,  power  or 
fuel,  prime  movers,  and  operating  environment). 

(5)  Specialized  operator  training  requirements. 

(6)  Performance  specifications,  including  restraints  or 
limitations,  coverage  capability,  data  accuracy,  format,  and 
unusual  data  reduction  considerations. 

(7)  Function  for  which  each  instrumentation  system  will  be 
employed. 

(8)  Data  requirements  being  met. 

(9)  Concept  of  employment. 

(10)  Pretest  check  out  criteria  and  procedures. 
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e.  The  ITTS  support  plan  describes  the  function  for  which 
each  instrumentation  system  will  be  employed,  the  data 
requirements  being  met,  and  the  concept  of  employment.  It  also 
provides  pretest  check  out  criteria,  procedures  and  employment 
schedules  for  each  instrumentation  system. 

f.  Planning  for  instrumentation  support  for  an  operational 
test  is  the  responsibility  of  the  operational  tester.  He 
appoints  a  Chief  of  Instrumentation  to  deal  with  specific 
requirements.  The  Chief  of  Instrumentation  is  part  of  the  test 
directorate  and,  as  such,  reports  to  the  Test  Director.  His 
responsibilities  include  the  following: 

(1)  Responsible  for  operations,  maintenance,  collection,  and 
control  of  all  test  instrumentation. 

(2)  Coordinate  with  the  Chief  of  Data  Management  to  insure 
smooth  flow  of  instrumented  data  to  and  through  the  data 
management  process. 

(3)  Serve  as  a  member  of  the  DAG  to  provide  instrumentation 
expertise. 

5-72.  Documentation  of  the  ISP 

Planning  for  ITTS  support  is  an  iterative  process  which  begins 
with  general  ITTS  needs  and  evolves  into  a  specific  ITTS  design. 

a.  The  central  document  for  ITTS  planning  is  the  ISP.  It  is 
prepared  by  the  ITTS  threat  analyst  and  instrumentation  engineer. 
They  are  usually  primary  members  of  the  task  force  planning  the 
test  and  evaluation. 

b.  The  purpose  of  the  ISP  is  to  insure  a  concept  which 
satisfies  test  "^enarios  and  data  requirements  and  provides  bases 
of  understand!  between  testers,  evaluators,  and  the  ITTS 
personnel  cone  ing  test  support.  It  provides  information  on 
performance  rec  rements,  preliminary  design,  and  impacts  of  ITTS 
systems . 

c.  The  ISP  ensures  a  smooth  transition  from  the  ITTS 
developers  to  the  ITTS  users.  Detailed  instructions  for 
development  of  an  ISP  are  provided  in  figure  5-5. 

(INSERT  FIGURE  5-5) 

d.  This  ITTS  planning  must  begin  early  in  the  test  planning 
cycle  to  insure  timely,  adequate  instrumented  data  during  test. 
The  process  is  iterative,  beginning  with  general  requirements  for 
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ITTS,  then,  evolving  into  specific  ITTS  requirements  and  ITTS 
purchase  or  design  recommendations. 

5-73.  Instrumentation  Support  Concept 

This  section  of  the  ISP  describes  the  ITTS  concept  and  is 
composed  of  the  subsections  below. 

5-74 .  Purpose 

This  subsection  describes  the  purpose  which  is  to  provide  the 
basis  for  ITTS  development,  procurement,  installation  and 
testing.  This  plan  provides  structure  and  guidance  for 
collecting  large  and  detailed  amounts  of  test  data,  through  more 
extensive  and  precise  ITTS. 

5-75.  Scope 

This  subsection  describes  the  scope  and  should  introduce  topics 
to  be  discussed  later  in  detail  such  as  equipment,  training,  data 
collection,  scheduling,  and  number  and  placement  of  ITTS. 

5-76.  Approach 

This  subsection  describes  the  overall  approach  for  ITTS  support 
to  include  stating  data  requirements,  specifying  types  and 
numbers  of  instruments,  and  setting  up  data  collection  schedules. 
Most  importantly,  the  approach  should  address  the  concept  of 
balancing  the  realism  of  a  test  with  the  importance  of  data 
collection  since  maximum  collection  of  data  content  could  be  made 
at  the  expense  of  operational  realism  of  a  test. 

5-77.  Brief  System  Description 

This  subsection  should  include  a  concise  description  of  the 
system  undergoing  the  test,  its  components,  and  its  mission,  as 
well  as  any  special  requirements  it  may  create  for  testing. 
Additionally,  modifications  or  interfaces  of  ITTS  to  the  system 
under  test  need  to  be  defined.  This  description  should  also 
include  any  impact  or  effect  of  these  interfaces  on  the  tested 
system's  operation. 

5-78.  Instrumented  Data  Collection 

Reliable  operational  testing,  requiring  the  utmost  realism,  may 
conflict  with  collection  of  data.  So,  there  must  be  a  compromise 
which  maximizes  the  combined  issues  of  realistic  testing  and 
thorough  data  collection.  Increasing  the  efficiency  as  well  as 
the  accuracy  of  data  collection  while  minimizing  the  interference 
with  player  operations  is  the  major  goal  of  instrumentation. 

5-79.  Instrumentation  Data  Requirements 

The  ISP  specifies  data  requirements  as  described  in  the  TEP. 

This  will  include  special  considerations  for  some  systems.  Using 
this  information,  the  Plan  will  describe  the  types  of 
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instrumentation  necessary  to  facilitate  the  collection  of  the 
data . 

5-80.  Collection  Considerations 

In  planning  the  collection  process  a  number  of  special 
considerations  for  instrumentation  must  be  addressed.  The 
considerations,  of  course,  will  vary  from  system  to  system  since 
each  system  has  unique  problems  and  characteristics. 

5-81.  Goal 

A  goal  for  collection  of  instrumented  data  should  be  specified  in 
this  section.  This  goal  should  be  determined  with  due 
consideration  given  to  instrument  failures,  transmission  errors, 
or  other  such  unusual  conditions  that  may  develop.  A  realistic 
percentage  of  total  data  generated  must  be  developed  to  insure 
proper  analysis  can  take  place.  As  a  minimum,  critical  items ^ 
must  be  identified  for  collection,  to  include  a  minimum  quantity 
of  each.  In  some  cases  the  test  event  may  be  a  non-repeatable, 
singular  event,  in  which  case  it  becomes  a  critical  item  to 
collect,  therefore  redundant  collection  methods  or  systems  should 
be  planned.  In  other  cases,  the  evaluator,  analyst,  tester,  and 
instrumenter  must  insure  enough  data  is  collected  to  have  the 
proper  sample  size  for  conduct  of  the  designated  analytical 
procedure.  This  percentage  should  be  determined  well  in  advance 
of  the  test,  identified  with  a  value,  and  be  considered  in  both 
instrumentation  planning  and  test  event  scheduling. 

5-82 .  Redundancy 

Depending  on  the  particular  system  and  test,  there  may  be  a 
requirement  for  redundant  collection.  Redundant  instrumentation 
involves  more  than  one  instrument  or  more  than  one  type  of 
instrument  collecting  the  same  data  at  the  same  -time.  This  might 
involve  setting  up  dual  sensors,  for  instance,  if  the 
characteristics  of  the  system  make  many  cycles  of  testing 
infeasible.  This  redundancy  lowers  the  probability  of  missing 
pertinent  data  due  to  instrument  or  personnel  malfunction. 
Redundancy  must  be  balanced  against  cost  and  occurrence.  That 
is,  if  an  event  in  an  operational  test  calls  for  a  singular, 
destructive  event  (e.g.,  destroying  a  vehicle),  the  redundancy 
should  be  extremely  high. 

5-83.  Movement 

The  ISP  addresses  specific  considerations  of  instrumentation 
movement.  If  and  when  instrumentation  cannot  be  moved  without 
major  support  requirements,  that  fact  must  be  considered  in 
planning  events  and  in  coordination  with  the  test  planner.  This 
section  should  answer  questions  such  as: 
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a.  Does  the  operational  test  require  instrumentation  to 

move? 

b.  How  much  mechanical,  chemical  or  electrical  stress  on 
instrumentation  will  the  test  create? 

c.  How  much  stress  can  the  instrument  sustain  without 
malfunctioning? 

d.  Will  the  movement  reduce  the  instrument's  accuracy  to 
any  extent?  Can  some  of  this  movement  be  avoided? 

e.  How  long  will  the  movement  last? 

f.  Should  data  collectors  follow  the  movement? 

g.  Is  the  instrument  the  most  viable  candidate  for  data 
collection  desired  considering  the  movement  involved  in  the  test? 

h.  Will  the  instrumentation  need  to  be  ruggedized,  and, 
therefore,  how  much  additional  lead  time  will  be  required? 

5-84.  Additional  Equipment. 

The  ISP  should  also  discuss  the  issue  of  whether  additional 
support  equipment  will  be  necessary  to  make  the  instrumentation 
operable.  This  could  include  separate  transportation  required 
for  some  instrumentation.  As  a  minimum,  the  plan  should  include 
a  list  of  all  additional  equipment  required  with  explanations  of 
planned  use  and  a  list  of  the  corresponding  instrumentation  with 
which  the  equipment  will  be  used.  The  cost  of  instrumentation 
must  include  cost  of  the  additional  support  requirements  when 
determining  cost  trade-offs  with  manual  data  collection. 

5-85.  Interface  with  ADP 

The  ISP  requires  close  and  continuous  coordination  with  the  Data 
Management  Plan  and  the  ASP  to  insure  compatibility.  This 
section  includes  plans  to  provide  appropriate  interfaces  between 
the  instrumentation  system (s)  and  the  ADP  system (s)  which  will 
aid  in  the  streamlined  transfer  of  data  from  the  instrumentation 
equipment  to  the  ADP  files  or  data  bases.  These  procedures 
should  specify  the  techniques,  protocols,  and  formats  for 
accomplishing  the  transfer  of  data  and  plans  for  verifying  that 
complete  transfer  has  occurred. 

5-86.  ITTS  Requirements 

The  ISP  will  detail  requirements  for  a  particular  system  by 
addressing  the  following  issues: 
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a.  Number  and  Type  of  Evaluated  Systems.  The  test  designer 
decides  the  number  of  evaluated  systems  to  be  instrumented  in  the 
test.  He  makes  his  decision  respecting  feasibility,  cost,  and 
risk.  All  the  aspects  of  threat  portrayal,  data  requirements, 
accuracy,  and  tolerances  must  be  considered.  Normally,  all 
systems  under  test,  including  baseline  systems,  are  instrumented 
but  do  not  necessarily  require  target  or  threat  simulator  systems 
on  a  one-to-one  basis. 

b.  Data  Inputs.  This  section  of  the  plan  describes  data 
input  points,  such  as,  type  and  location.  It  also  details  how 
data  inputs  will  be  recorded  and  details  any  security 
requirements  necessary  for  the  data  (e.g.,  encryption).  This 
section  should  include  any  other  special  considerations  of  data 
inputs  unique  to  a  particular  system  which  may  require 
instrumentation.  For  instance,  the  ARSIS  instrumentation  on  the 
Howitzer  Improvement  Program  (HIP)  must  tap  into  the  HIP’S  own 
data  bus  for  some  pieces  of  data. 

c.  Mobility.  This  section  describes  and  details  the  mobility 
requirements  and  considerations  for  the  ITTS.  Though  test 
procedures  for  mobility  are  implied  in  the  description  of  the 
system  operation,  these  mobility  factors  should  be  discussed 
thoroughly  here.  Concepts  such  as  travel  time,  type  of  travel, 
extent  of  travel,  operability  during  travel,  and  continuity  of 
operations  should  be  considered  for  instrumentation.  In  some 
cases  instrumentation  may  be  attached  to  a  system  under  test,  but 
in  other  cases  it  may  need  prepositioning.  If  prepositioned, 
then  mobility  and  the  capability  to  move  from  one  site  to  another 
become  major  considerations  in  allocation  of  ITTS  resources. 

d.  Scenarios.  The  various  scenarios  of  the  operational  tests 

should  be  described  in  relation  to  the  ITTS  which  is  needed.  The 
scenario  should  include  the  extent  of  intelligence  on  the  enemy, 
all  staging  and  props  needed  to  set  up  the  circ  tances  for 
testing  the  system  in  a  realistic  environment  (  nuclear, 

biological,  or  chemical),  and  the  test  requirem  ls  accompanying 
that  environment.  Threat  simulators,  especially  in  the  area  of 
electromagnetic  propagation,  have  a  major  scenario  impact  upon 
ITTS  support.  Scenarios  should  provide  details  on  the 
presentation  of  targets  to  detection,  target  acquisition,  and 
firing  systems. 

e.  Security.  A  security  plan  for  the  ITTS  must  be  developed 
to  insure  adequate  physical  security  and  communication  security, 
if  required.  If  collected  data  is  classified,  TEMPEST  or  other 
security  methods  are  to  be  followed.  DOD  range  security  requires 
that  all  radio  frequency  data  signals  be  protected.  Storage, 
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transfer,  and  accountability  procedures  for  equipment  must  be 
described  in  detail. 

f.  Power  Sources.  Power  sources  for  ITTS  should  be 
determined  and  provided  if  not  readily  available.  Power  failures 
as  well  as  power  surges  must  be  anticipated.  Care  should  be 
taken  to  insure  that  the  equipment  voltage  capacity  and  available 
voltages  match,  or  that  transformers  are  available.  If 
non-standard  power  requirements  are  necessary,  incorporation  of 
these  requirements  into  the  OTP  is  critical.  Requirements  for 
redundancy  of  power  sources  should  also  be  specified. 

g.  Collection  Locations.  This  section  specifies  the 
locations  of  data  collection  points  and  insertion  point  for 
targets  and  threat  simulators.  It  should  include  a  brief 
description  of  type  and  extent  of  data  to  be  collected  or 
insertions  to  be  made  at  each  point. 

(1)  Clear  instructions  should  be  provided  for  each  collection 
point  to  insure  that  each  point  is  manned  as  necessary  and  is  in 
the  proper  location.  A  communication  plan  should  be  developed 
which  would  allow  for  readiness  checks  at  each  point  of 
collection. 

(2)  Of  course,  some  points  will  be  located  on  the  system 
being  tested,  and  some  will  be  stationary. 

(a)  The  advantages  of  instrumentation  being  attached  to  the 
system  under  test  are  that  the  collection  process  is  made  more 
flexible,  and  the  tests  are  usually  more  realistic.  The  primary 
disadvantages,  however,  are  the  interaction  and  probable  impact 
of  instrumentation  on  the  normal  operation  of  the  system. 

Another  disadvantage  is  that  attached  instrumentation  takes  great 
stress  and  strain,  and  therefore  accuracy  might  be  lost. 
Furthermore,  attached  instrumentation  cannot  be  as  elaborate,  and 
requires  more  support  planning  for  computer  linkages  and  other 
logistical  problems. 

(b)  The  advantages  of  stationary  instrumentation  correspond 
to  the  disadvantages  of  mobile  instrumentation.  For  instance, 
stationary  instrumentation  can  be  more  elaborate,  and  have  high 
performance  sensors;  however,  in  using  this  stationary 
instrumentation,  location  or  other  constraints  may  be  placed  on 
testing,  therefore  losing  realism  in  the  test. 

h.  Soldier  Monitoring.  Soldiers  representing  typical  user 
troops  (players)  will  be  monitored  during  some  tests.  Details  of 
this  monitoring  effort  need  to  be  stated  clearly  in  the  ISP,  as 
well  as  referenced  in  the  Data  Management  Plan.  The  ISP  should 
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detail  how,  if  applicable,  personnel  are  instrumentally 
monitored,  in  what  phases,  and  the  effect.  For  example,  video 
footage  of  a  process  may  be  appropriate  in  some  cases,  while  in 
others  monitoring  of  some  specific  bodily  function  (e.g.,  heart 
rate)  may  be  appropriate.  Instrumentation  for  system  operator 
actions  which  measure  timeliness,  efficiency,  and  effectiveness 
should  also  be  described  here. 

i.  Candidate  Systems.  This  subparagraph  should  list  all 
potential  ITTS  systems  for  OT  of  an  acquisition  system  and  should 
contain  a  brief  description  including  its  potential  specific  use. 
It  should  include  a  priority  list  of  the  best  ways  to  acquire 
these  systems,  e.g. ,  use  of  existing  Government-owned  systems, 
lease  of  commercial  items,  purchase  of  commercial  items,  and 
fabrication  of  items. 

(1)  This  information  must  be  developed  early  in  the  ISP  draft 
development  so  negotiations  for  purchase  or  development  planninq 
or  paperwork  may  be  initiated  early. 

candidate  systems  should  include  a  statement 
of  feasibility.  A  particular  system  may  be  the  best,  most 
desirable  instrumentation  system  for  some  data  collection,  but 
might  be  too  expensive  to  buy  or  lease,  or  it  might  require  too 
many  personnel  to  operate  relative  to  the  importance  of  the  data 
It  would  collect. 

(3)  In  the  final  analysis,  the  efficiency  and  cost  of 
instrumentation  must  be  compared  to  efficiency  and  cost  of  a 
manual  data  collection.  As  the  ISP  becomes  finalized,  the 
selected  instrumentation  system  becomes  an  integral  part  of  the 
plan. 

5-87.  Other  Support  Requirements 

Addressed  in  this  section  "e  other  requirements  supporting  ITTS 
which  have  not  been  previ  ly  mentioned  in  the  plan.  These 
include  requirements  for  .  rage  and  handling  of  ITTS;  for 
transportation  of  ITTS  to  id  from  the  test  site  (e.g.,  special 
secure  transportation  required) ;  for  support  personnel  to 
transport,  handle,  maintain  and  store  ITTS;  for  special 
facilities  to  house  and  operate  ITTS;  for  support  communications 
necessary  for  ITTS;  and  for  special  tools  or  service  necessarv 
for  ITTS.  ^ 

5-88.  Test  Support 

This  section  describes  methods  for  implementing  the  ITTS  planning 
during  the  phases  leading  up  to  an  operational  test.  As  a 
minimum  it  should  include  milestones,  demonstrations,  training 
pilot  test,  and  OTRR.  ^ 
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5-89.  Milestones 

The  ISP  test  support  section  should  state  specific  milestones  for 
ITTS  (e.g.,  target  date  to  investigate  types  of  equipment,  target 
date  to  acquire  equipment,  target  date  to  install  equipment,  and 
target  date  to  train  personnel  to  use  equipment) . 

5-90.  Demonstrations 

Demonstrations  of  ITTS  should  be  conducted  within  the  framework 
of  the  larger  Test  Planning  Milestone  Schedule.  The  target 
dates,  extent,  and  number  of  demonstrations  will  vary  from  system 
to  system;  however,  some  general  guidelines  are  helpful.  A 
demonstration  should  probably  precede  each  OTRR.  Also,  if  a  new 
piece  of  ITTS  is  introduced  to  the  test  planning  relatively  late 
in  the  pre-test  schedule,  and  after  all  other  ITTS  has  been 
demonstrated,  it  should  not  be  overlooked.  A  date  should  be  set 
to  demonstrate  this  new  piece.  Finally,  a  request  by  one  of  the 
users  (e.g.,  data  manager,  test  director,  evaluator)  to  see  some 
piece  of  ITTS  demonstrated  again  should  be  granted  if  at  all 
feasible.  In  other  words,  demonstrations  should  be  performed  at 
planned  dates  as  well  as  any  additional  ones  deemed  necessary 
during  pretest  process.  In  early  test  planning  stages  ITTS  may 
be  demonstrated  by  itself;  however,  at  some  time  before  the  Pilot 
Test,  the  ITTS  must  be  demonstrated  on  the  tested  system. 

5-91.  Training 

Specific  dates  should  be  scheduled  for  training  personnel  in  the 
operations  of  ITTS.  The  amount  of  training  will  vary  according 
to  the  complexity  and  amount  of  ITTS,  but  most  of  the  training 
sessions  should  take  place  before  the  last  demonstration  and  last 
OTRR. 

5-92.  Pilot  Test 

By  the  date  of  the  Pilot  Test  as  scheduled  in  the  Detailed  Test 
Plan,  ITTS  should  be  in  full  operation  with  fully  trained 
personnel.  For  the  Pilot  Test  the  ITTS  should  be  in  place, 
operated  as  if  running  the  actual  test,  and  should  generate  real 
data  for  examination. 

5-93.  OTRR 

Each  OTRR  should  include  a  review  of  ITTS  progress  to  date. 

a.  At  the  first  OTRR,  270  days  before  the  test,  the  review  of 
ITTS  will  be  preliminary  and  general.  The  tentative  schedule  for 
ITTS  milestones  should  be  reviewed.  It  might  also  suggest  types 
and  extent  of  ITTS  anticipated.  If  ITTS  is  being  built,  a  status 
report  of  development  progress  must  be  made  near  the  time  of  the 
first  OTRR,  to  insure  that  the  equipment  will  be  ready  in  time 
for  test. 
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b.  At  the  second  OTRR,  60  to  90  days  before  the  test,  the 
review  of  ITTS  should  update  the  ITTS  schedule  and  should  detail 
the  ITTS  concept.  By  no  later  than  the  second  OTRR,  the  ISP 
should  also  cover  whether  some  previously  planned  ITTS  is  now 
deemed  unnecessary  and  whether  some  ITTS  needs  to  be  added. 

Also,  any  noticeable  problems  with  any  particular  ITTS  should  be 
addressed.  As  a  result  of  the  review,  a  decision  is  made  whether 
to  begin  training  personnel. 

c.  At  the  third  OTRR,  just  after  the  Pilot  Test,  review  ITTS 
one  final  time  and  from  data  collected  during  the  Pilot  Test, 
uncover  any  previously  hidden  problems  or  deficiencies  remaining 
in  ITTS.  The  review  should  also  make  an  assessment  of  whether 
personnel  are  completely  trained  and  are  aware  of  contingency 
plans.  This  should  also  include  a  discussion  of  how  well  each 
ITTS  system  works  in  conjunction  with  the  other  instruments  and 
interfaces  with  ITTS  systems,  and  if  there  is  a  problem  with 
cohesiveness,  what  solutions  should  be  explored.  Base  this 
discussion  on  the  last  two  or  three  ITTS  demonstrations  as  well 
as  on  the  Pilot  Test. 

5-94.  Conduct  of  Test 

During  the  actual  conduct  of  the  test  the  major  emphasis  for  ITTS 
is  on  properly  executing  the  plan  and  on  insuring  the  data 
collected  through  ITTS  flows  smoothly  into  the  data  management 
system.  The  coordination  of  ITTS  is  accomplished  within  the  test 
directorate  under  the  ultimate  control  of  the  Test  Director.  The 
chief  of  the  ITTS  organizational  element  is  an  integral  member  of 
the  test  directorate  working  for  the  Deputy  Test  Director  for  OT 
and  in  close  coordination  with  the  chief  of  data  management  and 
the  chief  of  the  ITTS  support.  This  requires  the  primary  chief 
of  ITTS  to  be  present  at  the  test  site  whenever  instrumented  data 
is  critical  to  the  overall  data  collection  procedure. 

5-95.  Special  ITTS  Considerations 

Sot  complex  ITTS  involve  software  which  must  be  developed.  This 
sc  are  usually  requires  long  lead  time  for  development,  it  is 

coi  Jy,  and  it  must  be  validated  well  before  the  commencement  of 
testing  to  insure  data  elements  are  collected  as  required  and 
extraneous  data  is  not  added.  The  ISP  should  address  these 
considerations,  and  should  alert  test  planners  to  the  long  lead 
time  and  costly  development. 


Section  VIII 

Audiovisual  (AV)  Support  Planning 
5-96.  AV  Support  Defined 
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Audio  and  visual  documentary  coverage  of  how  operational  testing 
was  conducted,  as  well  as  collection  of  selected  data  elements 
for  incorporation  into  the  report.  The  AV  documentation  covers 
the  testing  and  other  tester  activities  to  include  test  unit, 
test  directorate,  side  tests/demonstrations,  instrumentation, 
data  collection,  data  processing,  and  reports/briefings. 

5-97 .  Need  for  AV  Support 

Each  OTE  iteration  requires  an  AV  Support  Plan  (AVSP)  which 
results  in  adequate,  effective,  and  timely  AV  support  for  testing 
to  include  data  collection,  reduction,  analysis,  and  reporting. 
The  purpose  of  AV  support  is  twofold.  First,  AV  serves  as  a 
documentary  of  how  testing  was.  conducted,  what  the  test  sites 
looked  like,  how  effective  data  collectors  were,  etc.  AV  is  also 
used  to  collect  actual  test  data  as  an  input  and  integral  part  of 
the  report.  Using  AV  in  this  manner  is  particularly  useful  in 
the  realm  of  human  factors  analysis. 

5-98 .  Determining  Requirements  for  AV  Support 

The  concept  for  AV  support  is  jointly  prepared  by  the  AV  support 
personnel  and  the  test  officer  in  coordination  with  the  evaluator 
and  describes  the  support  to  be  provided  in  order  to  meet  the  AV 
requirements  of  the  test  and  the  evaluation.  It  permits  quick 
identification  and  resolution  of  requirement  misinterpretation 
prior  to  test.  If  identified  requirements  cannot  be  met,  the 
issues  are  raised  to  appropriate  levels  for  resolution.  The 
operational  tester  determines  the  extent  of  documentary  AV 
necessary  and  the  AV  test  data  desired.  He  develops  specific  AV 
plans  accordingly.  General  requirements  for  these  two  types  of 
AV  support  should  evolve  through  the  test  planning  cycle  into 
specific  requirements,  and  be  recorded  in  the  AVSP. 

5-99.  Documentation  of  AV  Support  Planning 

This  section  provides  guidance  for  incorporation  of  AV  support 
planning  for  OTE.  This  planning  results  in  development  of  the  AV 
support  paragraph  in  the  TEP,  the  AV  Support  Concept  Appendix  to 
the  TEP,  and  the  AV  Support  Plan  Appendix  to  the  DTP.  This 
section  applies  to  all  OT.  Tests  of  major  systems,  with  large 
data  bases  and  highly  instrumented  data  collection,  may  require 
roost  of  the  elements  in  the  plan.  Tests  which  are  less  complex 
may  not  require  all  parts  of  the  plan,  however  the  topics  cans  be 
used  as  planning  guides  to  insure  adequate  consideration  of  these 
areas  is  included  in  test  planning. 

5-100.  AV  Support  Planning  Process 

Planning  must  provide  necessary  AV  support  to  OTE  data  management 
requirements  and  provide  narrative  description  of  test  reports 
and  briefings.  This  section  explains  how  support  planning 
occurs,  and  discusses  details  of  how  to  develop  and  write  support 
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plans  for  AV  support.  The  section  describes  plans  in  detail  but 
allows  flexibility  for  the  tester  to  tailor  the  plan  for  his 
specific  test.  The  test  planner  describes  the  plan  to  support 
the  test  using  videotape,  still  photography,  or  other  audiovisual 
means.  The  plan  proposes  interim  reports,  schedules  field 
coverage,  and  sets  deadlines  for  each  phase  of  the  preparation 
and  approval  of  a  final  videotape. 

5-101.  TEP  AV  Support  Concept 

This  optional  appendix  to  the  TEP  contains  details  of  planned  AV 
support  not  contained  in  the  main  body  of  the  document.  It  is 
expanded  into  the  AVSP  in  the  DTP.  The  concept  has  no  fixed 
format,  but  using  the  order  of  presentation  given  for  the  AVSP 
Plan  provides  complete  coverage  of  the  subject  and  permits  easy 
expansion  from  the  concept  to  the  plan. 

5-102.  AV  Scheduling 

Development  of  an  implementation  schedule  is  paramount  to  an 
AVSP.  This  schedule  must  be  in  consonance  with  the  overall  test 
schedule  to  allow  adequate  planning  and  development  time  for 
support  of  the  test.  The  support  plan  is  the  responsibility  of 
the  tester  with  input  from  the  evaluator.  The  tester,  in  turn, 
assigns  a  AV  specialist  with  requisite  technical  expertise  to 
help  prepare  the  plan. 

5-103.  Preparation  of  the  AVSP 

When  AV  support  is  required  for  data  collection  and  test 
documentation,  the  requirements  are  prepared  by  the  tester  and 
supporting  AV  element.  They  describe  the  support  to  be  provided 
in  order  to  meet  the  AV  requirements  of  the  OT.  Advance  planning 
permits  quick  identification  and  resolution  of  any  requirements 
misinterpretation  prior  to  the  test.  The  support  needed  is  based 
on  AV  requirements  developed  during  overall  test  planning.  The 
AVSP  is  the  plan  to  support  the  test  using  videotape,  still 
photography,  or  other  audiovisual  means.  It  proposes  ini  im 
reports,  schedules  field  coverage,  and  sets  deadlines  fo  ach 
phase  of  the  preparation  and  approval  of  a  final  videotaf 

a.  In  general,  color  videotape  and  color  negative  still 
photographic  coverage  are  to  be  collected  for  all  significant 
test  events,  beginning  with  training  and  continuing  until 
completion  of  the  final  phase. 

b.  Enough  AV  material  is  to  be  collected  to  support  a 
narrative  description  of  test  issues  and  purpose,  each  type  test 
event,  the  conditions  under  which  the  test  was  conducted,  and  how 
data  were  collected  and  stored.  In  addition  to  the  general  test 
documentation,  coverage  of  test-specific  subject  areas  (such  as 
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mobility,  detectability,  engagement  sequences,  and  MOPP  4  trials) 
is  suggested. 

c.  For  each  such  subject  area,  the  number  and  type  of  shots 
are  described  -  '•video  and  still  shots  will  be  taken  of  the  test 
system  being  rigged  for  LAPES  delivery,  followed  by  shots  of 
delivery  and  any  damage  sustained."  Planned  shooting  schedules 
are  generally  provided  in  terms  of  days  or  weeks,  on  site,  and 
with  points  of  contact  for  additional  intermittent  support. 

5-104.  Documentation  of  the  AVSP 

Planning  for  AV  support  is  an  iterative  process  which  begins  with 
general  AV  needs  and  evolves  into  a  specific  AV  design. 

a.  The  central  document  for  AV  planning  is  the  AVSP.  It  is 
prepared  by  the  tester. 

b.  The  purpose  of  the  AVSP  is  to  insure  a  concept  which 
satisfies  data  and  documentation  requirements  and  provides  bases 
of  understanding  between  testers,  evaluators,  and  the  AV 
personnel  concerning  AV  test  support. 

c.  The  AVSP  ensures  a  smooth  transition  from  the  tester  to  AV 
operators.  The  writer  of  the  AVSP  should  clearly  distinguish 
requirements  for  each  type  of  AV  requirement  in  the  plan. ^ 
Detailed  instructions  for  development  of  an  AVSP  are  provided  in 
figure  5-6. 

(INSERT  FIGURE  5-6) 

5-105.  AV  Concept  in  the  AVSP 

This  section  of  the  AVSP  describes  the  AV  concept  and  is  composed 
of  the  subsections  detailed  below. 

5-106.  Purpose 

This  subsection  states  the  purpose  which  is  to  provide  complete 
visual  documentation  of  testing  procedures  and  results.  It  also 
serves  to  support  a  narration  or  report  on  all  aspects  of  testing 
from  training  to  the  final  phases,  to  include:  test  objectives, 
testing  site,  test  conditions,  and  procedures  of  data  collection. 

5-107.  Scope 

This  subsection  should  describe  the  scope  and  should  outline 
topics  such  as  the  amount  of  collection  of  visual  material 
required  throughout  the  testing  process  to  portray  accurately 
each  phase  of  testing  and  all  results.  It  involves  planning 
specific  shots,  specific  equipment,  specific  personnel. 
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This  subsection  should  describe  the  overall  approach  to 
audiovisual  planning  which  hinges  on  test  schedule.  The  approach 
should  contain  topics  such  as  planning  shootings  considering 
timeframes  and  test  deadlines  with  general  instructions  as  to 
what  shots  are  desired,  and  yet  allow  the  schedule  to  be  flexible 
enough  to  adjust  to  unexpected  occurrences  in  testing,  equipment, 
or  personnel. 

5-109.  Support  Description 

This  subsection  contains  a  summary  of  planned  audiovisual  test 
support,  and  provides  direction  for  specific  issues.  This 
discussion  should  include  significant  events,  personnel  and 
training,  and  procedures. 

a.  Significant  Events.  This  paragraph  should  list  all  events 
or  issues  for  which  audiovisual  coverage  is  deemed  necessary. 
Though  the  significant  events  may  vary  from  test  to  test,  the 
following  provides  guidelines  for  typically  relevant  events.  It 
should  list  each  significant  event,  including  details  of  every 
aspect  of  the  event . 

(1)  Detectability.  Videotaping  this  specific  subject  gives 
an  indication  of  how  detectable  the  weapon  system  is  in  the  eyes 
of  the  enemy.  This  involves  positioning  an  audiovisual  shooting 
downrange  from  the  system  at  a  vantage  point  realistically 
similar  to  the  position  of  the  enemy. 

(2)  Tactical  Mobility.  Videotape  should  also  be  made  from 
vantage  points  which  portray  the  ease  and  speed  with  which  the 
system  becomes  and  remains  operative. 

(3)  Engagement  Sequences.  This  refers  to  uninterrupted  video 
footage  of  a  complete  operational  sequence,  from  the  point  at 
which  the  gunner  detects  a  target  ur  11  one  round  of  fire  has 
been  completed,  for  example.  This  uence  should  be  shot  in  its 
entirety  from  many  different  angles  that  every  action  taken  by 
the  gunner  and  every  response  by  the  -ystem  can  be  recorded  for 
study.  This  engagement  sequence  may  also  include  set-up  and 
tear-down  processes  for  systems  which  require  these  actions. 

(4)  MANPRINT.  The  human  factors  analyst  should  be  consulted 
as  to  what  photographic  footage  would  be  appropriate  to  portray 
human  factors  in  testing.  This  determination  should  be  made 
concurrent  with  the  development  of  human  factors  issues.  The 
Automation  Support  Plan  should  include  a  schedule  for  preparing 
the  soldiers  and  test  site. 

(5)  Still  Layouts.  Still  photos  should  be  taken  of  each 
separate  component  of  the  system,  and  of  the  entire  system. 
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These  can  be  made  in  a  studio  with  a  plain  shadowless  backdrop. 
These  photos  can  be  for  general  reference  and  as  an  aid  to  a 
briefing  or  report. 

(6)  Targets.  Coverage  of  live  fire  tests  should  be  taken  of 
all  target  types  which  the  system  engages,  showing  system  rounds 
hitting  and  missing.  Closeup  photography  should  depict  effects 
of  hits  on  the  target,  damage  sustained,  and  other  lethality 
effects. 

b.  Personnel  and  Training.  This  paragraph  should  include 
details  of  type  and  number  of  personnel,  and  type  and  amount  of 
training  required  to  accomplish  all  audiovisual  aid.  This 
training  may  simply  consist  of  practicing  the  various  shots  which 
will  be  required,  and  discussing  with  the  audiovisual  personnel 
the  schedule  of  test  events,  and  any  test  peculiarities  which  he 
may  encounter.  Personnel  and  training  for  audiovisual  test 
support  will  differ  from  system  to  system,  so  the  procedures  for 
providing  and  training  personnel  must  be  flexible.  For  instance, 
a  system  requiring  high  redundancy  or  difficult  test  conditions 
would  require  more  audiovisual  training  and  maybe  more  full  time, 
on-site  personnel;  whereas,  a  system  test  which  is  less  complex 
or  less  lengthy  could  better  adjust  to  fewer,  less  trained 
audiovisual  personnel. 

c.  Procedures.  This  paragraph  should  explain  the  procedures 
that  audiovisual  personnel  should  follow  for  audiovisual  support 
in  testing.  It  should  address  all  foreseeable  problems.  Some 
examples  are:  the  procedures  for  procuring  audiovisual  equipment, 
the  procedure  for  developing  film,  and  the  procedures  for 
developing  an  audiovisual  package  of  production  quality  (for 
presentations  and  briefings) ,  complete  with  narration. 

5-110.  Operational  Requirements 

The  AVSP  should  detail  the  requirements  for  AV  support  of  the 
operational  test.  The  plan  should  expand  on  the  requirements  to 
include  specific  instructions  for  carrying  out  the  requirements. 

5-111.  Equipment 

a.  Video.  In  most  cases,  use  of  videotape  is  desired  over 
still  photography.  Video  coverage  provides  a  more  continuous  and 
cohesive  view  of  specific  action.  Video  is  paramount  in  the 
complete  engagement  sequence  in  order  to  assess  overall 
effectiveness  and  suitability  from  start  to  finish. 

b.  Still.  Still  photos  serve  primarily  as  a  supplement  to 
the  video  presentation.  They  are  most  useful  in  specific  shots 
of  each  component  of  the  system  for  extensive  examination  and  for 
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damaged  or  defective  systems  and  components.  Still  photos  allow 
for  carefully  planned,  detailed  shots,  from  many  different 
angles. 

5-112.  Subject  Areas  for  Support 

Included  in  the  AVSP  should  be  a  section  covering  specific 
audiovisual  requirements  for  the  two  major  areas  of  testing, 
operational  effectiveness  and  operational  suitability.  The  plan 
should  detail'  AV  support  necessary  for  recording  effectiveness 
and  suitability  information.  Provide  general  guidelines, 
explanations,  and  expectations  for  the  AV  support  to  each  subject 
area  to  include: 

a.  Type  of  support  required  (video  or  still,  limited  coverage 
or  continuous  coverage) . 

b.  Quantity.  Number  of  cameras  and  personnel  needed  to 
obtain  required  footage. 

c.  Location.  The  optimum  locations  of  the  cameras  to  obtain 
the  required  information. 

d.  Plan.  Complete  description  of  the  integrated  plan  for  the 
AV  coverage. 

5-113.  Scheduling 

Scheduling  of  audiovisual  support  consists  of  three  parts  which 
correspond  to  the  three  phases  of  the  test:  pretest,  pilot  test, 
conduct  of  the  test.  The  Audiovisual  Support  Plan  should  contain 
audiovisual  schedules  for  each  of  these  three  phases. 

a.  Pretest.  The  plan  should  specify  aspects  of  pretesting 
which  need  to  be  photographed,  such  as:  test  site,  equipment, 
terrain,  and  training. 

b.  Pilot  3t.  Full  scale  collection  of  footage  should  be 
made  of  the  :>t  test  and  procedures  so  that  video  can  then  be 
examined.  Tr  >  will  allow  for  refinements  in  the 
instrumentaticii,  the  ADP  equipment,  and  in  the  testing  process 
itself  before  full  scale  testing.  A  schedule  should  be  set  so 
that  the  video  will  be  prepared  for  a  quick  look  presentation. 

c.  Conduct  of  Test.  The  AVSP  should  map  out  general 
scheduling  of  all  pertinent  test  events,  training,  and 
procedures.  It  should  specify  times,  places  and  extent  of 
coverage  desired,  plus  any  special  considerations.  It  should 
also  specify  the  number  of  photographers  for  each  event,  type  of 
film  and  type  of  photographic  equipment  (video  or  still) .  The 
plan  should  include  a  description  of  any  other  photographs  which 
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may  be  required  during  conduct  of  the  test  such  as  maintenance 
incidents  or  MANPRINT  and  safety  problems.  However,  it  should  be 
written  such  that  it  will  allow  flexibility  for  schedule  changes. 
The  video  should  again  be  prepared  for  a  quick  look.  Then  the 
plan  should  set  a  schedule  for  developing  the  video  into  a  full 
presentation  with  narration  which  will  be  viewed  by  the  approval 
authority  for  the  Test  Report. 

5~114.  Special  Requirements  and  Continuous  Coverage 
This  section  of  the  AVSP  will  describe  any  special  requirements 
for  audiovisual  support,  such  as,  tests  at  night  or  tests  under 
extreme  cold  or  heat. 

5-115.  Non-Standard  Tests 

Given  the  peculiarities  of  operational  testing  from  system  to 
system,  the  Audiovisual  Support  Plan  should  describe  any 
anticipated  non-standard  testing  procedures  to  allow  audiovisual 
personnel  to  be  prepared  for  them. 

5-116.  Limited  Resources 

Audiovisual  resources  may  be  limited  to  some  extent  at  particular 
sites. ^  The  Support  Plan  in  this  section  should  prepare  the 
audiovisual  personnel  for  these  deficiencies,  so  that  appropriate 
adjustments  may  be  made  to  allow  for  full  audiovisual  support. 

5-117.  Support  Requirements 

This  part  of  the  plan  will  detail  any  specific  instructions 
available,  such  as  degree  of  continuous  coverage.  Also 
considered  in  this  section  are  requirements  for  support  from 
local  audiovisual  agencies,  both  commercial  and  government. 

5-118.  Contingencies 

Contingency  plans  will  be  detailed  here  which  provide  guidance 
for  audiovisual  personnel  in  the  event  that  testing  does  not  go 
as  originally  planned.  This  makes  the  audiovisual  concept  well 
structured  and  more  flexible.  The  plan  may  include  contingencies 
for  the  following: 

a.  Equipment  Failure.  In  the  event  of  audiovisual  equipment 
failure  a  contingency  plan  might  include  directions  for 
replacement  or  backup  equipment,  alternative  equipment,  how  and 
where  to  procure  it,  any  special  operating  procedures,  and  lead 
time  necessary  to  acquire. 

b.  Changing  Requirements.  Sometimes  as  testing  proceeds, 
plans  and  schedules  change.  New  tests  are  added,  planned  tests 
are  restructured,  some  tests  are  dropped  altogether,  or 
requirements  for  data  and  audiovisual  collection  may  change.  The 
Audiovisual  Support  Plan  should  include  a  contingency  plan  which 
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will  direct  audiovisual  support  if  requirements  do  change  during 
the  testing  process.  This  contingency  should  include  a  guide  as 
to  which  test  officials  should  be  consulted  if  changes  occur, 
time  necessary  to  adjust  for  changes,  and  specific  procedures  to 
follow  in  order  to  allow  testing  to  go  forward  without  losing 
audiovisual  support. 


Section  IX 

ADP  Support  Planning 


5-119.  ADP  Support  Defined 

Input,  storage,  computing,  control,  and  output  services  of  test 
data,  using  electronic  circuitry  within  a  computer  system  to 
perform  arithmetic  and  logical  operations  by  means  of  internally 
stored  or  externally  controlled  programmed  instructions 

5-120.  Need  for  ADP  Support 

Each  OTE  iteration  requires  an  Automation  Support  Plan  (ASP) 
which  results  in  adequate,  effective,  and  timely  automation 
support  for  testing  to  include  data  collection,  reduction, 
analysis,  and  reporting. 

5-121.  Determining  Requirements  for  ADP  Support 
The  concept  for  ADP  support  is  jointly  prepared  by  the  computer 
analyst  and  the  test  officer  and  describes  the  support  to  be 
provided  in  order  to  meet  the  automation  requirements  of  the 
test.  It  permits  quick  identification  and  resolution  of 
requirement  misinterpretation  prior  to  test.  If  identified 
requirements  cannot  be  met,  the  issues  are  raised  to  appropriate 
levels  for  resolution. 

5-122.  Documentation  of  ADP  Support  Planning^ 

This  section  provides  guidance  for  incorporation  of  ADP  support 
planning  for  OTE.  This  planning  results  in  development  of  the 
ADP  support  paragraph  in  the  TEP,  the  Automation  Support  Concept 
Appendix  to  the  TEP,  and  the  Automation  Support  Plan  Appendix  to 
the  DTP.  This  section  applies  to  all  OT.  Tests  of  major 
systems,  with  extremely  large  data  bases  and  highly  instrumented 
data  collection,  may  require  most  of  the  elements  in  the  plan. 
Tests  which  are  less  complex  may  not  require  all  parts  of  the 
plan,  however  the  topics  cans  be  used  as  planning  guides  to 
insure  adequate  consideration  of  these  areas  is  included  in  test 
planning. 

5-123.  ADP  Support  Planning  Process 

Planning  must  provide  necessary  ADP  support  to  OTE  data 
management  requirements.  This  section  explains  how  support 
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planning  occurs,  and  discusses  details  of  how  to  develop  and 
write  support  plans  for  ADP  support.  The  section  describes  plans 
in  detail  but  allows  flexibility  for  the  tester  to  tailor  the 
plan  for  his  specific  test. 

5-124.  TEP  Automation  Support  Concept 

This  optional  appendix  to  the  TEP  contains  details  of  planned  ADP 
support  not  contained  in  the  main  body  of  the  document.  It  is 
expanded  into  the  ASP  in  the  DTP.  The  concept  has  no  fixed 
format,  but  using  the  order  of  presentation  given  for  the  ASP 
Plan  provides  complete  coverage  of  the  subject  and  permits  easy 
expansion  from  the  concept  to  the  plan. 

5-125.  ADP  Scheduling 

Development  of  an  implementation  schedule  is  paramount  to  an  ASP 
plan.  This  schedule  must  be  in  consonance  with  the  overall  test 
schedule  to  allow  adequate  planning  and  development  time  for 
support  of  the  test.  The  support  plan  is  the  responsibility  of 
the  tester  with  input  from  the  evaluator.  The  tester,  in  turn, 
assigns  an  analyst  with  technical  expertise  to  help  prepare  the 
plan. 

5-126.  Preparation  of  the  ASP 

When  automation  support  is  required  for  data  management,  the 
requirements  are  prepared  by  the  tester  and  supporting  management 
information  systems  (MIS)  element.  They  describe  the  support  to 
be  provided  in  order  to  meet  the  ADP  requirements  of  the  test. 
Advance  planning  permits  quick  identification  and  resolution  of 
requirements  misinterpretation  prior  to  the  test.  The  support 
needed  is  based  on  automation  requirements  dev:jloped  during 
overall  test  planning  and  may  include: 

a.  A  problem  definition  including  identification  of  data 
types,  data  volume,  processing  requirements,  performance 
requirements  (turnaround,  response  times,  etc.),  ADP  security, 
and  reporting  requirements  (hard  copy,  online  queries,  etc.). 

b.  A  summary  of  the  automation  system  concept. 

c.  A  technical  description  of  the  data  flow. 

d.  Equipment  including  computers,  printers,  interfaces, 
decoders,  and  communications. 

e.  Software  required  (system  application  specific  commercial 
packages  and  noncommercial  programming  requirements) . 

f.  Data  dictionary  including  element  specification  (size, 
type,  etc.)  and  data  element  definition. 
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g.  Milestones  (for  MIS,  analysts,  and  other  users) . 

h.  Personnel  requirements  (programmers,  systems  analysts, 
data  entry) . 


5-127.  Documentation  of  the  ASP 

Planning  for  ADP  support  is  an  iterative  process  which 

with  general  ADP  needs  and  evolves  into  a  specific  ADP  design. 

a.  The  central  document  for  ADP  planning  is  the  ASP. ^  It  is 
prepared  by  the  ADP  systems  analyst.  He  is  usually  a  primary 
member  of  the  task  force  planning  the  test  and  evaluation. 


b.  The  purpose  of  the  ASP  is  to  insure  a  concept  which  ^ 
satisfies  data  requirements  and  provides  bases  of  understanding 
between  testers,  evaluators,  and  the  ADP  systems  analyst 
concerning  automated  test  support.  It  provides 

performance  requirements,  preliminary  design,  and  impacts  of  adp 
•  systems . 

c.  The  ASP  ensures  a  smooth  transition  from  the  ADP  developer 
fproqrammer)  to  the  ADP  user  (normally  the  data  manager). 

Detailed  instructions  for  development  of  an  ASP  are  provided  in 
figure  5-7. 


(INSERT  FIGURE  5-7) 

5-128.  ADP  Support  Concept 

This  section  of  the  ASP  describes  the  ADP  concept.  As  an  initial 
step,  the  ADP  analyst,  in  coordination  with  the  Data  Manager  for 
the  test  and  the  evaluator,  develops  an  automation  support 
concept  which  gives  a  summary  of  the  system.  This  first  step 
provides  the  basis  for  the  initial  part  of  the  plan.  This  concept 
should  be  written  for  non-ADP  personnel,  an^  hould  describe  the 
ADP  system  with  respect  to  organization,  fl  of  data,  and  method 
of  operation  that  the  system  ADP  analyst  is  i  quired  to  follow  in 
order  to  develop  the  system.  The  ADP  concept  development  begins 
the  iterative  process  for  ADP  planning,  which  is  refined  and 
expanded  as  the  TEP  is  further  refined.  The  concept  is  composed 
of  the  subsections  described  below. 

5-129.  Organization  for  Automation  Support 
This  subsection  should  detail  the  organization  for  ADP. 
organization  for  ADP  is  highly  interrelated  with  the  Data 
Management  organization;  however,  the  ADP  analyst  has  the  primary 
responsibility  for  automation  support  planning.  The  ^DP  analyst 
is  responsible  for  keeping  the  T&E  task  force  informed  of  the  ADP 
progress.  Additionally,  he  is  responsible  for  development  of  the 
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required  software,  operation  of  the  equipment  for  storing  data, 
and  maintenance  and  control  of  the  data  base  on  the  system. 

5-130.  Data  Flow  and  Anticipated  Volume  of  Data 
This  subsection  should  outline  the  expected  flow  of  data  and 
provide  a  graphic  figure  showing  the  flow.  Figure  5-8  depicts  an 
example  of  the  flow  of  generated  data  from  its  entry  as  either 
automated  or  manual  to  its  output  as  reports  and  data  displays. 
All  data  from  operational  tests  will  not  necessarily  follow  all 
of  the  steps,  but  the  main  points  of  entry,  reduction,  temporary 
storage,  validation,  permanent  loading  into  data  base,  and 
reports  resulting  from  the  data  base  will  generally  be  used  for 
an  OT  of  any  size.  These  steps  should  be  addressed  as  well  as 
the  anticipated  volume  of  data  to  be  received  and  stored  for  the 
particular  OT. 

(INSERT  FIGURE  5-8) 

5-131.  Method  of  Operation 

This  subsection  should  describe  as  completely  as  possible  the 
method  of  operation.  The  ASP  is  written  as  many  of  the  documents 
and  events  leading  to  a  test  are  still  in  the  conceptual  or 
planning  stages;  so,  pertinent  assumptions  of  the  method  of 
operation  that  may  affect  the  overall  automation  support  should 
be  included  in  the  plan.  These  assumptions  eventually  are 
replaced  by  decisions  as  the  concepts  evolve  into  actual 
demonstrations.  As  a  general  rule,  assumptions  should  be  made 
for  any  area  of  uncertainty.  Assumptions  may  be  classified  in 
two  major  categories:  Equipment  or  Services  and  System  or 
Software. 

a.  Equipment  or  Services.  Included  in  this  category  are  any 
ADP  assumptions  concerning  hardware  or  personnel.  Examples  of 
areas  that  may  require  assumptions  are:  availability  of 
computers  by  type  and  number,  telecommunication  lines  leadtime, 
qualification  of  data  entry  personnel  or  programmers, 
availability  of  ADP  personnel  at  test  site,  and  security 
capability  of  equipment. 

b.  System  or  Software.  Included  in  this  category  are  any 
assumptions  concerning  the  overall  operating  environment  of  the 
automated  system.  Examples  are:  type  and  availability  of  the 
operating  system,  completion  and  check  of  any  new  software, 
availability  of  contractor  support  if  required,  and  utilization 
of  any  common  data  structure  such  as  the  OPTEC  OTERAM  system. 

5-132.  Design  Details 
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This  section  describes  the  ADP  system,  its  requirements,  and  the 
procedures  the  ADP  will  use  to  meet  the  requirements  for  ADP 
support . 

5-133.  ADP  Functional  Description 

This  subparagraph  of  the  ASP  should  describe  the  ADP  system  s 
functional  characteristics  and  operations  to  provide  a  worlcing 
knowledge  of  the  system.  It  should  include  an  exact  description 
of  how  the  system  operates,  the  path  of  the  data,  how  the  data 
will  be  stored,  and  how  and  to  whom  it  will  be  available. 

5-134.  Detailed  Characteristics 

This  section  presents  a  detailed  description  of  the  performance 
requirements  of  the  ADP  system  in  non-ADP  terminology,  including 
interface  with  other  elements  of  data  management.  Emphasis  is 
placed  upon  the  system  being  "user  friendly.”  System  failures 

not  unexpected  in  any  development,  and  this  section  addresses 
the  most  likely  failures  and  contingencies  to  overcome  them. 

5-135.  Performance  Requirements  ^ 

Minimum  performance  requirements  are  defined  and  specified. 

Where  available  these  tolerances  are  extracted  from  the  TEP  or 
otherwise  specified  by  the  evaluator.  In  some  cases  performance 
requirements  that  are  omitted  from  the  TEP,  but  common  to  ADP, 
should  be  emphasized  in  this  section.  The  performance 
requirements  may  be  subdivided  into  accuracy,  timing,  and  inputs. 

a.  Accuracy.  Every  effort  should  be  made  to  design  into  the 

ADP  system  the  means  to  insure  accuracy  and  veracity  of  the  data 
as  it  flows  through  the  system.  Several  levels  of  checks  can  be 
built  into  the  process  of  raw  data  collection  to  eliminate 
transcription  errors.  Individual  data  point  values  entered  into 
required  fields  can  be  checked.  Mutually  exclusive  elements  must 
coincide,  and  forms  agree  where  appropriate.  ^This  section 

describes  where  data  -  ry  checks  are  made  to  eliminate 
typographical  errors.  t  should  describe  where  to  expect  errors 
in  transmission  and  whe  t,  to  do.  Finally,  the  rounding  factors 
for  such  items  as  times,  distances,  and  meter  readings  should  be 
explained.  There  should  be  reference  to  the  corresponding 
paragraph  of  the  TEP  where  appropriate. 

b.  Timing.  The  timing  requirements  specified  in  the  TDP  for 
data  flow  and  turnaround  are  restated  in  this  section  of  the  ASP 
to  insure  proper  understanding  by  both  the  ADP  programmer  and  the 
tester.  Response  times  for  availability  of  ADP  output  should  be 
specified  (48  hours  is  a  good  rule  of  thumb) .  For  example,  "the 
time  requirement  for  the  period  from  the  time  an  incident  has 
completed  DAG  review  until  the  validated  data  is  accessible  in  a 
report  from  the  data  base  is  48  hours.”  Additionally,  the 
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response  time  of  the  system  should  be  described  here  to  include 
reasons  for  expected  delays  such  as  heavy  use  periods  or 
complicated  mathematical  operations. 

c.  Inputs  and  Outputs.  This  is  a  description  of  the  data 
inputs  and  outputs.  In  a  data  base  of  any  size  this  subparagraph 
should  give  a  basic  description  and  refer  the  reader  to  an 
appendix  where  each  element  is  listed.  It  is  related  to  and  may 
be  cross-referenced  to  the  Data  Base  Dictionary.  It  should 
include  the  following  data. 

(1)  Data  name:  A  generic  or  external  name  for  the  data 
element. 

(2)  System  name:  The  proper  name  used  in  the  data  base. 

This  is  the  one  that  must  be  used  when  working  with  the  data 
base. 

(3)  Width  (char) :  The  minimum  and  maximum  length  in 
characters  when  the  item  is  printed. 

(4)  Format:  The  type  of  data  in  the  element.  Analphabetic, 
N=numeric,  AN=alphanumeric,  Y=year. 

(5)  Edit  checks:  Specifies  the  edit  checks  the  system  will 
perform. 

(6)  Source:  The  original  source  of  the  data. 

5-136.  Failure  Contingencies 

The  ADP  systems  analyst  would  be  remiss  not  to  plan  for  system 
failures.  There  are  two  general  approaches  to  failure 
contingencies:  backup  and  fall  back. 

a.  Backup.  Ideally,  every  ADP  system  would  be  backed  up 
several  times,  but  this  must  be  considered  in  light  of  cost, 
time,  and  risk.  A  significant  amount  of  redundancy  should  be 
planned,  especially  in  the  number  of  microcomputers  or  terminals. 
If  a  host  system  is  used,  rerouting  or  alternative  systems  may  be 
considered.  Since  communication  failure  is  a  common  occurrence, 
especially  in  electrical  storms,  alternative  lines  should  be 
planned.  Planning  for  failure  of  personnel  to  perform  must  also 
be  included  when  analyzing  redundancy  requirements  since 
operators  may  be  absent  for  a  myriad  of  reasons. 

b.  Fall  back.  The  more  serious  failure  is  the  type  for  which 
redundancy  has  not  or  could  not  be  planned.  Several  options  need 
to  be  planned,  especially  for  backing  up  working  data  on  a 
regular  basis.  For  most  operational  tests,  microcomputers  will 
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store  enough  on  diskette  to  bridge  a  communication  failure. 

Power  failures  could  cause  catastrophic  results  if  not 
anticipated;  so,  procedures  such  as  storing  data  by  alternative 
means  (e.g.,  diskette,  tape)  should  be  considered. 

5-137.  Data  Entry 

There  are  two  basic  formats  for  entering  data  into  an  ADP  system; 
manual,  and  automatic  feed.  The  extent  to  which  each  of  these 
methods  will  be  used  in  the  data  processing  of  the  test  should  be 
expressed  in  the  ASP. 

a.  Manual.  Manual  entry  occurs  when  data  is  generated  from 
forms  and  questionnaires,  usually  measuring  RAM  data  or  human 
factors  data.  Forms  and  questionnaires  should  be  standardized 
and  made  as  simple  and  objective  as  possible  in  order  to 
facilitate  the  entry  of  the  data  into  the  ADP  system. 

Furthermore,  some  type  of  quality  control  of  the  manual  data 
should  be  planned  to  insure  that  only  accurate  data  is  entered 
into  the  data  base,  and  that  no  typographical  errors  occur.  This 
section  should  describe  the  data  that  will  be  entered  by  hand, 
exactly  how  it  will  be  entered,  which  personnel  will  check  it, 
and  which  personnel  will  enter  it. 

b.  Automated.  Data  is  entered  by  automation  when  it  has  been 
generated  and  collected  by  instruments  which  are  tied  directly 
into  the  ADP  system,  or  collected  and  stored  on  some  ADP  medium. 
Here,  a  quality  control  plan  should  be  devised  to  check  the 
linkage  between  the  instrumentation  and  the  ADP  system.  For  this 
type  of  entry,  the  plan  should  describe  the  data,  the  path  it 
will  take,  where  and  how  it  will  be  controlled,  and  which 
personnel  will  control  it. 

5-138.  Data  Processing 

Thif'  next  step  in  the  processing  of  data  may  consist  of  loading 
dat  into  storage  or  into  an  organized,  readable  form. 

Prc jedures  for  this  task  should  be  included  in  the  ASP,  if 
applicable. 

5-139.  Data  Verification/Validation 

The  ASP  should  specify  the  procedures  for  verifying  and 
validating  the  data  being  loaded  into  the  ADP  system.  This 
includes  answers  to  where,  when,  how  and  by  whom  these  tasks 
would  be  accomplished. 

5-140.  Quality  Control 

This  part  is  an  overall  view  of  the  control  of  the  ADP  system, 
its  components,  its  support,  and  the  data  it  processes.  The  ASP 
should  contain  specifics  of  this  control. 


5-48 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


5-141.  Storage 

While  data  is  being  validated  by  the  data  manager,  it  will  be 
stored  in  special  temporary  files  which  cannot  be  processed  by 
the  users.  After  validation,  the  data  will  be  uploaded  into  the 
data  base  of  the  host  computer  for  permanent  storage. 

5-142.  Data  Manipulation 

In  some  cases  the  form  of  the  data  may  need  to  be  altered  before 
it  can  be  shipped  to  another  data  base,  or  be  summarized  and 
analyzed.  These  considerations,  along  with  the  details  for 
accomplishing  this  manipulation,  should  be  specified  in  the  ASP. 

5-143.  Output  Reports 

The  ASP  should  detail  how  the  data  will  be  reported.  Some  report 
considerations  are: 

a.  Test  Report.  This  report  is  the  main  document  stating  the 
processed  results  of  the  test. 

b.  Data  Displays.  The  ASP  should  specify  how  the  data  will 
be  displayed,  in  accordance  with  the  TEP.  Data  displays  should 
be  identified  and  demonstrated  as  early  as  possible  in  the 
process  to  insure  capability  to  present  data  in  conjunction  with 
the  evaluator's  planned  TER. 

c.  Other  Preliminary  Reports.  The  ASP  should  discuss 
provisions  for  other  preliminary  reports  of  testing  results  such 
as  DAG  reports,  Quality  Control  reports.  Scoring  Conference 
reports,  and  Test  Incident  Reports. 

5-144.  Analysis  Requirements 

The  plan  should  also  address  the  requirements  for  analysis  on  the 
data,  and  how  this  should  affect  its  processing. 

5-145.  External  Interfaces 

Included  in  the  ASP  should  be  a  statement  of  who  can  access  the 
data  base  and  from  where  outside  the  actual  testing  realm  (e.g., 
another  test  agency,  some  other  test  site,  or  members  of  the 
DAG)  . 

5-146.  Data  Base  Structure 

This  section  of  the  ASP  should  finalize  the  organization  and 
structure  of  the  test  data  base.  It  should  also  detail  the 
following  considerations: 

a.  Dictionary.  The  structure  of  the  dictionary  may  vary  from 
test  to  test  but  should  generally  contain  the  data  name,  the 
system  name,  the  data  description,  the  range  of  the  data,  and  the 
data  source. 


5-49 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992  (DRAFT) 

b.  Sample  Forms.  The  data  base  should  be  structured  using 
sample  collection  forms  to  record  RAM  factors  as  well  as  other 
factors  as  they  develop. 

5-147.  Output  Reports  .  t  i 

Considerations  for  output  reports  should  be  detailed  in  the  plan 
and  reflected  in  the  data  base  structure. 

5-148.  Environment 

This  section  of  the  ASP  should  specify  the  physical  and 
environmental  ADP  requirements  to  support  the  test.  Issues 
include: 

a.  Hardware.  The  ASP  should  list  all  hardware  inyolved, 
computer  name  and  type,  as  well  as  all  accessory  requirements  ^ 
such  as  a  modem.  Accessories  should  be  checked  for  compatibility 
with  the  computer.  The  ASP  should  also  discuss  procedures  for 
care  of  hardware.  For  instance,  hardware  should  be  housed  in  a 
temperature  controlled  facility  with  proper  electrical  current. 
Furthermore,  some  type  of  backup  electricity  should  be  planned 
and  made  accessible  in  the  event  of  power  failure.  The  ASP 
should  also  state  what  type  and  number  of  personnel  will  be 
required  for  operation. 

b.  communications.  Also,  the  ASP  should  list  communications 
equipment  required  and  a  description  of  how  the  communication 
network  will  work.  Both  internal  and  external  communication 
networks  should  be  considered.  Plans  for  hook-up  should  be  made 
well  in  advance  (at  least  eight  months)  of  the  commencement  of 
testing.  The  plan  should  tell  how  many  commercial  lines  and  how 
many  internal  lines  need  to  be  installed,  and  the  type  and  number 
of  modems  necessary.  This,  of  course,  will  vary  from  test  to 
test,  but  any  test  should  have  at  least  two  or  three  commercial 
lines  with  long  distance  capability.  The  plan  should  also 
specify,  as  soon  as  possible,  which  phone  company  will  be 
handling  the  network,  hook-up,  and  operations. 

c.  Facilities.  Requirements  for  facilities  are  covered  In 
other  test  planning  documents,  but  they  should  also  be  considered 
for  compatibility  with  ADP  equipment.  These  considerations 
include,  as  previously  mentioned,  temperature  controlled  rooms 
and  backup  electrical  power.  Furthermore,  depending  on  the 
nature  of  the  testing,  there  may  be  requirements  for  ADP  in  vans 
or  some  other  mobile  housing.  Also,  the  number  and  placement  of 
buildings  at  test  headquarters  may  affect  ADP  plans. 

d.  Cost.  A  breakdown  of  the  cost  of  each  ADP  element  should 
be  contained  here.  This  cost  should  be  weighed  against  the 
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complexity  and  detail  of  the  test  requirements  to  determine  if 
some  elements  are  not  cost  effective. 

5-149.  System  Development 

This  section  of  the  ASP  should  articulate  the  specific  supporting 
role  of  the  ADP  systems  analyst  to  the  test.  Topics  for 
discussion  should  include: 

a.  Software  Development.  The  discussion  should  define 
exactly  what  role  the  analyst  plays  in  the  development  of 
software  for  ADP.  It  should  tell  to  whom  the  analyst  reports, 
how  frequently,  and  how  many  support  personnel  (programmers)  will 
be  needed.  The  plan  should  provide  instructions  as  to  what 
equipment  is  available  and  should  give  a  time  schedule  for 
completion  of  the  development  of  the  software. 

b.  User's  Manual.  The  analyst  should  oversee  the  writing  of 
the  user's  manual.  The  ASP  should  specify  how  and  when  the  ADP 
ana.lyst  plans  to  proceed  with  the  writing  and  publication  of  the 
manual.  This  should  include  completion  dates  for  drafts  and 
final  copy.  Also  the  plan  should  specify  how  the  analyst  plans 
to  delegate  portions  of  the  writing  of  the  manual. 

c.  Coordination.  The  analyst  should  coordinate  all  phases  of 
the  ADP  system.  For  instance,  the  analyst  should  periodically 
update  all  personnel  involved  in  ADP  system  development.  The  ADP 
analyst  should  coordinate  all  meetings  and  activities  between 
programmers,  manual  writers,  personnel  who  will  operate  the 
equipment,  evaluators,  testers,  and  other  task  force  members. 

d.  Training.  The  analyst,  since  he  will  be  intricately 
involved  in  the  planning  and  development  of  the  ADP  system, 
should  also  oversee  the  training  of  personnel  to  operate  the 
equipment.  This  will  insure  that  smooth,  cohesive  and  proper 
training  occurs. 

5-150.  Test  Support 

The  ASP  should  also  contain  all  the  ADP  related  milestones 
necessary  to  support  the  test  plus  a  detailed  description  of 
each. 


a.  Milestones.  In  conjunction  with  overall  OT  milestones, 
the  ASP  should  contain  specific  milestones  for  ADP  planning. 

Among  these  are:  target  date  for  confirmation  of  ADP  equipment 
list,  target  date  for  software  development  and  user's  manual,  and 
target  date  for  deployment  of  equipment. 

b.  Demonstrations.  Imbedded  in  the  calendar  for  ADP  support 
should  be  planned  demonstrations  of  ADP  equipment  and  software. 
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The  dates  before  test  (T-dates)  and  extent  of  these 
demonstrations  will  vary  from  test  to  test.  However,  there 
should  be  no  less  than  three.  Furthermore,  if  new  equipment  or 
personnel  enter  the  test  process,  additional  demonstrations 
should  be  scheduled  as  necessary. 

c.  Training.  As  specified  by  the  test  directorate  and  the 
ADP  systems  analyst,  specific  dates  and  procedures  should  be 
detailed  for  training  personnel  to  operate  ADP  equipment. 

Training  will  vary  according  to  the  complexity  of  the  equipment, 
but  training  should  be  completed  by  or  before  the  last  OTRR.  In 
the  event  of  software  or  program  changes,  or  hardware 
modifications,  personnel  may  need  to  be  trained  to  perform  these 
specialized  functions. 

d.  Pilot  Test.  By  the  date  of  the  scheduled  Pilot  Test,  the 
ADP  system  should  be  fully  operational  with  all  known 
deficiencies  corrected.  For  the  Pilot  Test  all  equipment  should 
be  tested  as  if  it  were  running  in  the  actual  test,  and  therefore 
should  accept  and  process  incoming  data. 

e.  OTRR.  As  a  part  of  each  OTRR,  there  should  be  a  section 
reviewing  the  ADP  planning  progress  to  that  point.  At  the  first 
OTRR,  approximately  270  days  before  the  test,  a  review  of  ADP 
planning  should  include  discussion  of  the  known  requirements  for 
the  processing  of  data,  the  feasibility  of  the  tentative 
schedule,  and  the  cost  feasibility  of  the  types  and  extent  of  ADP 
equipment  desired.  The  OTRR  should  also  review  the  progress  of 
communications  planning.  The  second  OTRR,  60-90  days  before  the 
test,  should  discuss  acquisition,  housing,  hook-up,  and  training 
for  the  ADP  system.  It  should  update  the  tentative  schedule  of 
events,  and  should  articulate  any  concerns  about  the  system.  The 
final  OTRR,  just  after  the  Pilot  Test,  should  review  the  data 
processing  from  the  Pilot  Test  and  confirm  that  the  system  is  ^ 
working  properly,  that  the  supportir  equipment  is  functioning  in 
conjunction  with  the  other  equipment  and  that  the  personnel  have 
been  fully  trained. 

f.  Control  and  Retention  of  Data  Base.  The  ASP  should  plan 
for  control  and  retention  of  the  data  base.  This  will  discuss 
who  has  access  and  how  to  access  the  data  base.  This  planning 
will  insure  that  the  data  base  is  not  altered  or  erased.  To 
accomplish  this,  the  plan  should  provide  guidelines  for  security 
against  tampering,  and  guidelines  for  power  failure, 
communication  breakdown  or  some  other  unexpected  happening. 


Section  X 

Planning  for  Test  Training 
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5-151.  Training  Plan 

The  training  concept  in  the  TEP  is  expanded  into  a  training  plan 
in  the  DTP.  The  training  plan  is  an  essential  element  of  the 
overall  test  planning  process.  Requirements  for  training  are 
developed  during  the  evolution  of  the  control  and  data  management 
plans.  For  each  category  of  personnel,  requirements  for  training 
are  identified.  A  program  for  eliminating  these  shortfalls  is 
then  developed.  The  training  plan  should  include  designation  of 
attendees,  lesson  plans,  examinations,  schedules,  performance 
tests,  designation  of  instructors,  and  location  of  instruction. 
Standard  Army  formats  for  these  documents  should  be  used.  There 
is  no  prescribed  format  for  a  Training  Plan. 

5-152.  Player  Personnel 

Training  of  players  will  provide  individual,  section,  and  unit 
level  skills  necessary  to  operate  and  maintain  the  equipment  or 
perform  the  tactics  to  be  tested.  The  results  of  the  test  will 
be  sensitive  to  the  amount  and  quality  of  player  training.  In 
fact,  for  some  tests  the  amount  and  type  of  training  received  by 
various  groups  of  players  will  be  a  major  test  variable.  For  all 
tests,  the  actual  training  received  should  be  documented  and  will 
become  a  part  of  the  final  report.  The  tester  must  receive  the 
trainer's  Operational  Test  Readiness  Statement  (OTRS)  before 
proceeding  with  testing. 

5-153.  Opposing  forces  (OPFOR) 

In  those  tests  where  OPFOR  is  included,  the  units  portraying  the 
enemy  must  be  trained  to  employ  the  tactics  of  the  threat  force. 
The  employment  of  U.S.  tactics  and  procedures  by  OPFOR  may 
invalidate  or  cast  doubts  on  results  of  such  a  test. 

5-154.  Controllers 

As  a  minimum,  controllers  will  be  completely  familiar  with  the 
control  plan  and  the  data  collection  plan.  They  must  be  capable 
of  making  sound  independent  decisions  on  control  matters  in  the 
absence  of  communications  and  should  be  able  to  act  as  data 
collectors  should  the  need  arise.  They  should  also  be  able  to 
implement  emergency  procedures. 

5-155.  Data  Collectors 

Training  for  data  collectors  will  include  instruction  on  the 
control  plan  and  the  data  collection  plan.  Particular  emphasis 
should  be  given  to  the  need  for  accuracy  and  unbiased 
objectivity.  Data  collectors  should  be  able  to  act  as 
controllers  and  implement  emergency  procedures,  if  necessary. 

5-156.  Data  Reduction  and  Quality  Control  Personnel 
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Training  for  data  reducers  and  quality  control  personnel  will 
provide  familiarity  with  the  data  reduction  plan  and  quality 
control  procedures.  Formal  instruction  is  usually  necessary,  and 
training  time  and  other  resources  must  be  planned  accordingly. 
Practical  exercises  with  substitute  data  will  provide  the  best 
preparation. 

5-157.  Test  Directorate  and  DAG  Personnel 

These  personnel  will  be  completely  familiar  with  all  aspects  of 
the  test.  Some  formal  instruction  is  usually  necessary ,  and 
training  time  and  other  resources  must  be  planned  accordingly. 


Section  XI 

Test  Support  Planning 


5-158.  Test  Support  Defined 

Test  support  addresses  the  categories  of  general  administrative, 
materiel,  and  service  support  necessary  for  the  performance  of 
overall  test  activities.  It  includes  general  support  functions 
to  support  both  field  testing  and  garrison  administrative 
requirements. 

5-159.  Need  for  Test  Support 

Each  OTE  iteration  requires  a  Test  Support  Plan  (TSP)  which 
results  in  adequate,  effective,  and  timely  test  support  for 
testing.  Test  support  encompasses  the  requirements  for  items 
including  but  not  limited  to;  equipment  (vehicles  including 
rental  cars,  generators,  heating  and  air  conditioning,  special 
fabrication  items;  locally  procured  equipment  such  as  copying 
machines  and  other  office  equipment,  military  standard  items); 
facilities  and  base  support  (including  offices,  shelters, 
latrines,  electricity,  telephones  and  other  communications 
equipment,  sic  ,  and  any  necessary  range  construction); 
supplies;  ammi  . tion;  and  fuel  (MOGAS,  JP— 4,  diesel).  Schedules 
are  developed  for  delivery  of  test  support  items  and  the  use  of 
facilities,  ranges,  and  equipment. 

5-160.  Determining  Requirements  for  Test  Support 
The  concept  for  test  support  is  jointly  prepared  by  test  support 
personnel  and  the  test  officer  and  describes  the  support  to  be 
provided  in  order  to  meet  requirements  of  the  test.  It  permits 
quick  identification  of  problems  and  resolution  of  requirement 
misinterpretations  prior  to  test.  If  identified  requirements 
cannot  be  met,  the  issues  are  raised  to  appropriate  levels  for 
resolution.  The  operational  tester  determines  the  extent  of 
support  necessary.  He  develops  specific  plans  accordingly. 
General  requirements  for  test  support  should  evolve  through  the 
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test  planning  cycle  into  specific  requirements,  and  be  recorded 
in  the  TSP. 

5-161.  Documentation  of  Test  Support  Planning 

This  section  provides  guidance  for  incorporation  of  test  support 
planning  for  OTE.  This  planning  results  in  development  of  the 
support  paragraphs  in  the  TEP,  the  Test  Support  Concept  Appendix 
to  the  TEP,  and  the  Test  Support  Plan  Appendix  to  the  DTP.  This 
section  applies  to  all  OT.  Tests  of  major  systems  may  require 
most  of  the  elements  in  the  plan.  Tests  which  are  less  complex 
may  not  require  all  parts  of  the  plan,  however  the  topics  cans  be 
used  as  planning  guides  to  insure  adequate  consideration  of  these 
areas  is  included  in  test  planning. 

5-162.  Test  Support  Planning  Process 

As  for  the  other  support  planning  requirements,  the  details  of 
test  support  requirements  and  planning  are  identified  and 
documented  during  planning  for  the  operations  and  data 
collection/reduction  requirements.  The  test  officer  will 
normally  have  a  test  support  officer  or  NCO  assigned  as  a  member 
of  the  test  team.  The  test  support  officer /NCO  coordinates  the 
documentation  of  the  test  support  requirements  and  formalizes  the 
requirements  in  the  test  support  plan.  Planning  must  provide 
necessary  test  support  to  the  test  site.  This  section  explains 
how  support  planning  occurs,  and  discusses  details  of  how  to 
develop  and  write  support  plans  for  tests.  The  section  describes 
plans  in  detail  but  allows  flexibility  for  the  tester  to  tailor 
the  plan  for  his  specific  test. 

5-163.  TEP  Test  Support  Concept 

This  optional  appendix  to  the  TEP  contains  details  of  planned 
test  support  not  contained  in  the  main  body  of  the  document.  It 
is  expanded  into  the  TSP  in  the  DTP.  The  concept  has  no  fixed 
format,  but  using  the  order  of  presentation  given  for  the  TSP 
provides  complete  coverage  of  the  subject  and  permits  easy 
expansion  from  the  concept  to  the  plan. 

5-164.  Test  Scheduling 

Development  of  an  implementation  schedule  is  paramount  to  an  TSP. 
This  schedule  must  be  in  consonance  with  the  overall  test 
schedule  to  allow  adequate  planning  and  development  time  for 
support  of  the  test.  The  support  plan  is  the  responsibility  of 
the  tester.  The  tester,  in  turn,  assigns  a  Test  Support 
Specialist  with  requisite  technical  expertise  to  help  prepare  the 
plan. 

5-165.  Preparation  of  the  TSP 

Test  support  requirements  are  prepared  by  the  tester.  They 
describe  the  support  to  be  provided  in  order  to  meet  requirements 
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of  the  OT.  Support  needed  is  based  on  requirements  developed 
during  overall  test  planning.  Planning  for  test  support  is  an 
iterative  process  which  begins  with  general  test  needs  and 
evolves  into  a  specific  support  design.  The  central  document  for 
planning  is  the  TSP  which  satisfies  and  documents  requirements 
and  provides  bases  of  understanding  between  testers,  evaluators, 
and  test  support  personnel.  Detailed  instructions  for 
development  of  an  TSP  are  provided  in  Figure  5-9. 


(INSERT  FIGURE 

5-9) 

5-166.  Test  Concept  in  the  TSP 

This  section  of  the  TSP  describes  the  support  concept  and  is 
composed  of  the  subsections  detailed  below: 

a. 

Purpose . 

b. 

Scope. 

c. 

Approach  to  Test  Support. 

d. 

Organization  for  Test  Support. 

5-167.  Logistics  Support  of  Test 

Logistics  support  is  generally  discussed  in  terms  of  six  areas: 

a.  Supply.  The  supply  requirements  and  schedules  for 
delivery,  storage,  and  distribution  of  all  types  of  expendable 
and  nonexpendable  supplies  required  to  support  the  test.  The 
supply  function  involves  the  acquisition  of  items  such  as 
cabinets,  typewriters,  and  support  vehicles.  Required  control 
and  accountability  procedures  are  described  (it  is  not  necessary 
to  reiterate  normal  supply  and  accountability  procedures  provided 
in  AR  735-11) . 

b.  Maintenance.  The  requirements  for  the  support  of  all 
administrative  equipment  to  be  used  in  the  test.  Maintenance  of 
the  test  item  is  performed  in  accordance  with  the  maintenance 
concept  developed  for  the  system  and  includes  management  of 
maintenance  operations,  facilities,  and  repair  parts  supply. 

c.  Transportation.  The  administrative  transportation  support 
of  the  test  directorate  and  related  activities  including 
transportation  for  player  units,  movement  of  the  test  item  and 
all  test  support  items  to  the  test  site,  inspecting  and  accepting 
these  items,  and  preparing  the  test  item  for  shipment  after  the 
test. 
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d.  Facilities.  The  requirements  for  the  use  of  real  property 
facilities  such  as  ranges,  maneuver  areas,  test  courses, 
airstrips,  railheads,  weather  stations,  office  space, 
communications,  dining  facilities,  recreation  facilities,  places 
of  worship,  and  storage  space.  Test-unique  construction  and 
maintenance  of  real  property  are  also  included. 

e.  Services.  The  requirements  for  local  services  to  include 
security  guards,  billeting,  garrison  and  field  rations,  clothing 
issue,  laundry  service,  and  special  services  for  troop  morale. 
Includes  the  self-service  supply  center  support  by  the 
installation. 

f.  Medical.  The  medical  support  plans  which  are  prepared  in 
coordination  with  the  responsible  command/staff  surgeon.  These 
plans  ensure  that  preventive  medicine,  treatment,  medical 
evacuation,  medical  supply,  and  emergency  communications  are 
available  to  adequately  support  the  test  directorate  and  player 
personnel.  Medical  and  health  considerations  associated  with  the 
tested  equipment  are  also  addressed. 

5-168.  Communications 

Planning  for  communications  consists  of  identification  of  the 
communications  systems  required  to  support  the  test.  Development 
of  the  control  and  data  management  plans  provide  the  basis  for 
communications  planning.  The  core  of  the  communications  plan  is 
development  of  radio  and  wire  nets  for  test  organization  control, 
players,  OPFOR,  controllers,  and  data  collectors.  Once 
requirements  are  determined,  the  net  designs  provide  the  basis 
for  requesting  equipment,  personnel,  and  frequencies. 

5-169.  Detailed  Cost  Estimate 

A  detailed  cost  estimate  is  revised  estimate  of  the  funds 
required  for  the  test.  The  format  is  the  same  as  that  used  in 
the  OTP.  Changes  in  the  estimate  from  the  OTP  version  should  be 
clearly  identified. 

5-170.  Safety 

The  safety  plan  section  describes  the  procedures  used  to  ensure 
safe  conduct  of  the  test  to  include  a  safety  briefing  (usually 
given  by  the  chief  controller)  to  all  player  and  test  directorate 
personnel  based  on  the  system  safety  release.  In  particular,  it 
identifies  required  safety  officers,  and  specifies  procedures  to 
ensure  that  operators  of  any  potentially  hazardous  equipment  are 
fully  trained.  Tests  posing  unusual  or  potentially  hazardous 
conditions  that  involve  risk  beyond  the  normal  call  of  duty 
require  approval  by  the  Surgeon  General  and  are  to  be  conducted 
within  the  provisions  of  AR  70-25.  The  system  safety  release  is 
also  included  or  referenced. 
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5-171.  Electronic  Warfare 

The  electronic  warfare  plan  is  a  detailed  set  of  instructions, 
procedures,  and  schedules  for  those  portions  of  the  test 
involving  electronic  warfare.  It  is  applicable  when  electronic 
warfare  is  a  major  condition  of  the  test  but  not  the  purpose  or 
major  objective. 

5-172.  Visitor  Control 

The  visitor  control  instructions  document  requirements  for 
visitor  control,  briefings,  security  clearance  verifications, 
conferences,  field  tours,  transportation,  special  equipment  (such 
as  laser  goggles,  boots,  and  field  pants),  and  other  related 
details. 

5-173.  Public  Affairs 

The  public  affairs  plan  describes  the  measures  to  process 
requests  or  actions  from  the  news  media  and  local  populace. 
Paragraph  3-29,  AR  360-5,  sets  the  Army  policy  on  media  coverage 
of  Army  test  activities.  It  states:  "The  Department  of  the  Army 
does  not  permit  media  coverage  of  developmental,  technology 
validation,  or  operational  testing  of  Army  systems.”  Consult 
with  the  local  public  affairs  officer  for  details. 

5-174.  OPSEC 

The  OPSEC  plan  addresses  the  sensitive  aspects  for  the  test,  the 
hostile  threat,  the  vulnerabilities,  countermeasures,  OPSEC 
training,  and  priorities  of  the  test.  If  appropriate  for  the 
test's  classification,  the  test  officer  must  plan  for  the 
protection  of  such  information  as  communications  (document  or 
electronic  means) ,  informative  patterns  and  signatures  (visual, 
acoustic,  electronic,  or  infrared) ,  and  stereotyped  procedures 
(tactical,  physical,  or  administrative) . 

5-175.  Security 

The  security  plan  encompasses  all  activities  r  aired  to  ensure 
internal  physical  security  at  the  headquarter.  rea  and  the 
security  of  the  test  site  along  with  their  assr ciated  property  to 
include  the  tested  system.  Coordinate  this  plan  with  your 
security  manager.  The  security  manager  should  also  be  consulted 
on  access  badges  and  security  clearances. 

5-176.  Long  Lead  Items 

The  necessity  for  long  lead  time  procurement  and/or  processing 
requirements  will  often  occur  in  the  test  process.  The  test 
officer  must  be  sensitive  to  the  early  identification  of  these 
requirements  and  to  identify  and  conduct  the  actions  necessary  to 
start  the  process  to  obtain  the  materiel  and/or  services 
required.  The  following  paragraphs  include  areas  of  potential 
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long  lead  time  items  which  should  be  considered  by  the  test 
officer  in  this  process. 

a.  Ammunition.  Will  special  ammunition  or  ordnance  be 
required? 

b.  Aircraft.  Will  tactical  and/or  special  purpose  test 
support  aircraft  be  needed? 

c.  Instrumentation.  Will  special  purpose  or  other 
instrumentation  be  used  that  involves  lengthy  procurement  details 
and  time? 

d.  Personnel  Skills.  Are  military  personnel,  contractors,  or 
civilians  with  unusual  skills,  qualifications,  or  training 
needed? 


e.  Equipment  Fabrication.  Do  mockup  targets,  support 
structures,  or  other  electromechanical  devices  need  to  be 
fabricated? 

f.  Contracts.  Are  advertised  or  negotiated  contracts 
necessary? 

g.  Environment.  Does  the  test  produce  environmental  concerns 
or  impacts  that  must  be  resolved  prior  to  test  start? 

h.  Target  Alteration.  Are  changes  required  in  existing 
equipment  configurations  or  tactical  aircraft?  Does  new 
equipment  need  to  be  installed?  If  so,  an  airworthiness  release 
must  be  obtained. 

i.  Permanent  Construction.  Will  there  be  any  permanent 
construction  requirements  or  alterations  to  existing  permanent 
base  and  range  facilities  (B&RF)? 

j.  Electrical  Power.  Do  firm  power  facilities  need  to  be 
installed? 

k.  Liquid  Petroleum  Gas/Water/ Septic  Tanks.  Are  liquid 
petroleum  gas,  water  lines  and  sources,  or  septic  tanks  required? 

l.  Nonstandard  Stock.  Are  there  any  demands  for  nonstandard 
or  nonstocked  supplies  of  equipment  items?  The  cost  and 
availability  must  be  determined  in  advance. 

m.  Rental  Car  Support.  Are  rental  vehicles,  vans,  or  other 
transportation  facilities  required? 
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n.  Telephone/FAX  Lines/Radios  (Communications).  What 
telephone  capability  (class  A,  DSN,  etc.),  facsimile  lines, 
radios,  or  other  communication  media  are  needed? 

o.  Computer  Support.  What  computers  for  large-scale  data 
collection  and  processing  are  required?  Are  PCs  for  onsite  or 
data  reduction  centers  needed? 


Section  XII 

Prerequisites  for  Test  Initiation 

5-177.  Operational  Readiness  Statement  (OTRS)  Requirements 

a.  Prior  to  the  start  of  the  test,  the  MATDEV,  CBTDEV,  and 
TNGDEV  each  provide  the  operational  tester  with  a  written 
statement  of  the  system's  readiness  for  OT.  The  operational 
tester  specifies  in  the  OTP  milestone  schedule  the  suspense  dates 
for  the  OTRS  (normally  90  days  prior  to  the  test  start  date) . 

b.  Deviations  from  the  required  readiness  standards  for  test 
(e.g.,  system  safety,  training)  require  a  statement  of 
explanation  by  the  OTRS  proponent  (MATDEV,  CBTDEV,  TNGDEV). 

c.  For  ACAT  I  and  II  OT,  conducted  in  support  of  a  MS  III 
production  decision  (beyond  LRIP  decision) ,  the  MATDEV  OTRS  must 
certify  that  the  system  is  ready  for  a  dedicated  phase  of  OTE 
(DODI  5000.2). 

d.  The  operational  evaluator  reviews  the  OTRS  to  ensure  that 
identified  deficiencies  will  not  affect  the  ability  of  the  OT  to 
answer  the  test  issues. 

e.  Prior  to  the  start  of  test,  the  operational  tester 
forwards  the  OTRS  to  the  appropriate  decision  authority  for 
information  or  resolution  f  disagreements  and  concerns. 

f.  For  operational  tests  not  conducted  by  OPTEC,  information 
copies  of  the  OTRS  are  provided  to  OPTEC.  An  OT  will  not  be 
initiated  until  all  OTRS  have  been  received  and  reviewed. 

5-178.  MATDEV  OTRS 

a.  The  MATDEV  describes  the  system  to  be  tested  in  terms  of 
size,  shape,  weight,  transportability,  and  functional 
characteristics . 

b.  For  software-driven  systems,  the  MATDEV  specifies  the 
software  version  to  be  tested  and  current  documentation  to  be 
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made  available.  A  detailed  statement  of  how  both  the  system 
hardware  and  software  characteristics  differ  from  a  fully 
representative  IOC  system  is  provided,  if  appropriate. 

c.  The  MATDEV  identifies  the  DT  objectives  which  have  been 
met  and  all  failures  and  deficiencies  which  have  been  corrected. 
Any  DT  objectives  not  met  or  failures  not  corrected  will  be 
detailed,  and  estimates  of  their  effect  on  OTE  described. 

d.  The  HATDEV  identifies  special  instrumentation  required  and 
the  availability  of  that  instrumentation  through  his  office. 

e.  The  MATDEV  identifies  the  system  maintenance,  training, 
and  supply  resources  requirements  which  are  to  be  evaluated 
during  test.  Military  resupply  procedures,  support  procedures 
and  special  support  requirements  are  defined.  If  contractor 
support  is  called  for,  the  specific  role  of  the  contractor  is 
defined  at  each  echelon. 

f .  The  MATDEV  estimates  the  current  and  projected  RAM 
performance  in  terms  of  the  system  ORD. 

g.  The  MATDEV  includes  a  detailed  statement  concerning  any 
restrictions  to  ordinary  operations  under  field  conditions  which 
will  exist  in  the  test. 

h.  The  MATDEV  provides  a  Safety  Release  for  the  system 
(obtained  from  TECOM)  or  identifies  the  status  of  the  release. 

5-179.  CBTDEV  OTRS 

The  CBTDEV  OTRS  verifies  that  the  doctrine,  organization,  threat, 
logistics  concept,  crew  drill,  and  standard  operating  procedures 
(SOPs)  in  the  CBTDEV s  support  packages  are  complete,  represent 
planned  employment,  and  are  approved  for  use  during  OT. 

5-180.  TNGDEV  OTRS 

The  TNGDEV  OTRS  verifies  that  the  training  concepts  and  materiel 
and  crew  drills  included  in  the  training  support  package  are 
complete,  representative  of  the  training  package  to  be  used  at 
fielding, ,  and  approved  by  TRADOC  for  use  during  OT.  In 
addition,  it  verifies  that  the  user  troops  have  satisfactorily 
complete  training  in  accordance  with  the  training  support  package 
and  are  ready  for  test. 

5-181.  Test  Unit  OTRS 

A  signed  OTRS  is  required  from  the  test  unit  commander.  This 
certifies  that  unit  personnel  are  MOS  qualified  through  use  of 
SQT,  CTT,  and  where  appropriate,  ARTEP  tasks.  This  statement 
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does  not  certify  that  unit  personnel  are  trained  on  the  test 
item. 

5-182.  Test  Support  Packages 

Test  support  packages  are  required  from  the  MATDEV,  CBTDEV,  and 
TNGDEV  prior  to  commencement  of  testing.  A  description  of 
support  packages  can  be  found  in  Chapter  3  and  Part  One  of  this 
pamphlet.  Milestones  for  submission  of  the  support  packages  are 
contained  in  the  TSARC-approved  OTP.  All  of  the  support  packages 
are  equally  important;  however,  development  and  delivery  of  the 
system  support  package  (SSP)  must  be  closely  monitored  because 
lack  of  a  complete  SSP  will  prevent  initiation  of  an  lOT  or  FOT, 
unless  a  waiver  to  proceed  is  obtained  lAW  AR  700-127.  The 
existence  of  an  incomplete  SSP  will  be  classified  as  a  critical 
test  incident  and  reported  by  message. 

5-183.  Safety  Release 

a.  A  written  system  safety  release  obtained  from  TECOM  must 
be  on  hand  prior  to  initiating  any  training  or  testing  involving 
user  troops.  The  test  officer  must  ensure  TRADOC  proponent 
schools  and  all  test  directorate/test  player  personnel  are 
knowledgeable  of  safety  precautions/procedures.  At  the  T-60 
OTRR,  the  program  sponsor  or  other  agency  responsible  for  the 
safety  release  will  provide  same  to  the  test  officer. 

b.  TECOM  normally  provides  safety  releases  for  hardware 
systems  on  a  reimbursable  basis.  The  program  sponsor  or  the  test 
requestor  (in  the  case  of  FDTE,  CEP  test,  and  customer  test)  is 
responsible  for  funding  TECOM  to  develop  the  safety  release  per 
AR  385-16.  Funding  for  the  safety  release  will  be  included  in 
the  OTP.  To  assure  timely  receipt  of  the  safety  release,  the 
operational  tester  must  proactively  coordinate  with  the  activity 
responsible  for  generating  the  safety  release. 

c.  For  NDI,  the  safety  release  is  provided  by  the  procuring 
agency  (MATDEV  or  the  CBTDEV) ,  as  appropriate.  In  all  cases,  the 
safety  release  is  provided  by  the  MATDEV  or  the  CBTDEV. 

d.  A  copy  of  the  safety  release  is  provided  to  the  commander 
of  the  organization  supplying  the  troops  to  ensure  that  the 
organization  is  informed  of  the  risks  identified.  For  weapon 
systems,  both  live  fire  and  non-fire  safety  releases  may  be 
required. 

e.  Where  appropriate,  the  safety  release  indicates  the 
results  of  The  Surgeon  General  (TSG)  investigation  of  medical  or 
health  problems  related  to  the  materiel  system  and  includes  a 
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certification  as  to  the  safety  of  user  troops.  User  test  using 
aircraft  require  an  airworthiness  release  (See  AR  70-62) . 

5-184.  TEP 

The  TEP  should  be  approved  as  described  in  Chapter  3.  If  the  TEP 
is  not  approved,  the  TEP  will  be  a  high  interest  item  for  the  T- 
60  OTRR.  Test  initiation  will  not  occur  without  an  approved  TEP. 

5-185.  Test  Limitations  and  Waivers 

The  tester  informs  the  independent  evaluator  of  each  and  every 
test  limitation  and  all  requests  for  waiver  to  the  test  program 
that  have  been  approved  by  the  appropriate  decision  authority. 


SECTION  XIII 

Operational  Test  Readiness  Review  (OTRR) 


5-186.  OTRR  Purpose 

OTRR  are  conducted  prior  to  each  OT  test  to  allow  Commander, 

OPTEC  (or  any  other  operational  tester)  to  assess  the  overall 
readiness  for  test  of  the  system.  The  OTRR  determines  readiness 
of  the  system,  support  packages,  instrumentation,  test  planning, 
evaluation  planning,  etc.  to  support  the  OT.  The  OTRR  includes 
identification  of  any  problems  which  impact  the  start  or  adequate 
execution  of  the  test  and  subsequent  evaluation  or  assessment  of 
the  system.  The  objective  of  the  review  is  to  determine  if  any 
changes  are  required  in  planning,  resources,  training,  equipment, 
or  timing  to  successfully  proceed  with  the  test. 

5-187 .  OTRR  Composition 

a.  OTRR  are  chaired  by  Commander,  OPTEC,  the  commander  of  any 
other  operational  test  activity  or  their  designees.  The 
Commander,  OPTEC  chairs  all  OTRR  for  ACAT  I,  ACAT  II,  MAISRC,  and 
DOD  Oversight  systems.  He  may  delegate  the  chair  for  a  specific 
OTRR.  OTRR  for  non-major,  non-oversight  systems  and  for  FDTE, 
CEP,  and  CT  will  be  chaired  by  Commander,  TEXCOM.  He  may 
delegate  the  chair  for  a  specific  OTRR. 

b.  Principal  OTRR  attendees  include  the  operational  tester, 
operational  evaluator,  MATDEV,  CBTDEV,  TNGDEV,  logistician, 
developmental  tester,  developmental  evaluator,  command  providing 
user  troops  for  test  (normally  FORSCOM) ,  HQDA  staff  elements, 
host  installation,  and  contractor. 

c.  The  operational  tester  (Test  Director  or  Test  Officer) 
will  be  responsible  for  planning,  administrative  support,  and 
reporting  results  for  the  OTRR.  For  full  evaluate  systems,  the 
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tester  works  in  close  coordination  with  the  operational  evaluator 
to  schedule  the  OTRR  and  establish  the  agenda. 

5-188.  OTRR  Schedule 

Three  OTRRs  are  essential  for  most  post-Milestone  I  OT.  When 
necessary,  any  of  the  participants  may  request  the  chair  convene 
an  additional  OTRR.  An  OTRR  may  not  be  used  for  purposes  which 
are  outside  its  intended  scope  such  as  system  reviews.  The  three 
essential  OTRR  are: 

a.  An  action  officer  level  review  (which  is  chaired  by  the 
operational  tester)  at  approximately  9  months  prior  to  test  (T- 
270) ,  held  in  conjunction  with  a  TIWG.  This  review  focusses  on 
identifying  those  activities  and  actions,  if  any,  that  appear  to 
be  moving  too  slowly  to  support  the  test  start  date  or  proper 
test  execution.  At  this  meeting,  any  misunderstandings  on  the 
identity  of  activities  responsible  for  elements  of  test  planning, 
readiness,  and  execution  are  resolved.  For  selected  high- 
interest  tests,  this  OTRR  may  be  elevated  to  a  GO-level  OTRR. 

b.  A  review  prior  to  resource  (player,  testers,  and 
equipment)  deployment  to  test  site  (normally  2  months  prior  to 
test  at  T-60) .  A  primary  consideration  of  this  review  is  to 
ascertain  if  any  known  problems  exist  which  would  delay  test 
start,  and  to  preclude  incurring  deployment  costs  when  the  test 
start  date  is  in  jeopardy.  At  this  review,  resource  providers 
confirm  their  readiness  to  release  the  resources  to  the  tester. 
MATDEV,  CBTDEV,  TNGDEV,  and  test  unit  OTRS  are  provided  to  the 
tester  at  this  review.  The  Safety  Release  should  be  provided  at 
this  OTRR,  but  must  be  provided  prior  to  beginning  of  hands  on 
training  of  test  players. 

c.  A  review  prior  to  the  beginning  of  record  test  in  order  to 
determine  if  the  tested  system,  players,  testers,  ITTS,  and  data 
reduction  procedures  are  ready  for  testing  for  record.  This  OTRR 
is  normally  conducted  at  the  test  site  during  latter  phases  of, 
or  immediately  following,  the  pilot  test.  In  addition  to  topics 
addressed  during  previous  reviews,  data  collection  and  data 
reduction  techniques,  functions  of  automatic  data  processing 
systems,  validity  of  pilot  test  data,  and  operations  of  the  DAG, 
if  appropriate,  are  examined.  The  test  officer  and  the  evaluator 
confirm  the  success  of  end-to-end  data  runs. 

5-189.  Pre-OTRR 

A  pre-OTRR  is  normally  conducted  the  day  prior  to  the  official 
OTRR.  The  pre-OTRR  is  an  action  officer  level  meeting  which 
attempts  to  reduce  known  problems  by  developing  solutions  and 
milestones  prior  to  the  OTRR.  Normally,  only  matters  that  could 
prevent  valid  testing  (potential  "shows toppers")  are  briefed  at 
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the  OTRR.  In  those  cases  where  the  T-270  OTRR  is  conducted  at 
the  action  officer  level,  there  is  no  need  for  a  pre-OTRR. 

5-190.  OTRR  Product 

The  resultant  product  of  each  OTRR  is  a  decision  by  the  chairman 
to  execute  the  T&E  as  planned,  to  direct  required  changes  to 
ensure  successful  test  execution,  or  to  recommend  (to  the  program 
decision  authority)  delay  or  cancellation  of  OT.  The  operational 
tester  (normally  OPTEC)  has  authority  to  suspend  start  of  testing 
if  readiness  of  the  materiel  or  IMA  system  offers  less  than  a 
reasonable  chance  of  meeting  established  criteria,  including 
posing  undue  hazards  to  personnel  or  equipment  (AR  73-1) .  OT  may 
also  be  delayed  when  it  becomes  evident  that  critical  test  data 
or  information  cannot  be  obtained  to  adequately  answer  the 
issues. 

5-191.  OTRR  Preparation 

a.  OPTEC  (or  other  OT  activity)  will  be  responsible  for 
scheduling  the  OTRR.  Attendees  will  be  notified  of  a  scheduled 
OTRR  and  the  planned  agenda  at  least  30  days  prior  to  the  review. 

b.  A  typical  OTRR  agenda  is  provided  at  figure  5-10.  It 
should  be  used  as  a  guide  in  developing  an  appropriate  agenda  for 
a  particular  system.  Mandatory  subjects  for  briefing  by  the 
tester  at  all  OTRRs  are  specifically  identified.  The  agenda 
should  always  include  provisions  for  the  MATDEV,  CBTDEV,  TNGDEV, 
and  test  unit  commander  to  provide  their  OTRS,  which  formally 
addresses  system  readiness  for  OT.  Additionally,  prior  to  OT  to 
support  the  Milestone  III  decision,  the  program  manager  certifies 
the  system  is  ready  for  a  dedicated  phase  of  OT. 

(INSERT  FIGURE  5-10) 

5-192.  OTRR  Documentation 

Minutes  of  an  OTRR  are  distributed  to  OTRR  participants  within  10 
working  days  after  adjournment  of  the  OTRR.  Within  3  working 
days  after  adjournment  of  the  OTRR,  external  commands  or  agencies 
are  notified  by  either  message  or  memorandum  of  any  issues  or 
problems  surfaced  during  the  OTRR  for  which  their  agency  has 
responsibility  for  resolving  prior  to  test  start.  The  message 
may  solicit  the  personal  assistance  of  the  agency  commander  in 
overseeing  necessary  corrective  actions.  Within  5  working  days 
after  adjournment  of  the  OTRR,  a  status  report  outlining  the 
results  of  the  OTRR  is  provided  to  the  appropriate  decision 
makers.  The  format  and  addressees  are  determined  on  a  case-by- 
case  basis  by  the  chairman,  based  on  the  outcome  of  the  review 
and  degree  of  assistance  required  to  resolve  outstanding  issues. 
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5-193.  Delay  or  Termination  of  Operational  Testing 

a.  In  the  event  that  an  OTRR  indicates  that  testing  should  be 
delayed  (i.e.,  inadequacies  of  SSPs,  OTRSs,  training,  test 
planning,  instrumentation,  etc.,  which  will  adversely  affect  test 
start,  execution,  or  its  realism  and/or  completeness) , 
alternative  courses  of  action  and  recommendations  are  developed 
which,  if  executed,  assist  in  maintaining  the  integrity  of  the 
test. 


b.  Due  to  one  year  notification  requirements  for  provision 
of  resources  for  support  of  OT,  a  seemingly  short  delay  in  the 
start  of  the  OT  result  in  a  delay  of  a  year  or  more  (see  AR 
15-38) . 


c.  If  a  determination  is  made  that  suspension  of  testing  is 
necessary,  the  chairman  expeditiously  forwards  the  issues  and 
recommendations  to  the  decision  authority,  with  information 
copies  to  the  MDR  principals,  for  a  decision  to  start,  delay,  or 
terminate  the  test. 


Section  XIV 
Pretest  Activities 


5-194.  General 

These  activities  involve  all  pretest  training,  organizing  for 
execution  and  support,  preparation  of  equipment  and  test  areas, 
the  pilot  test,  adjustment  of  plans  (if  necessary) ,  and  all  other 
actions  required  to  prepare  for  the  test.  The  training  plan  and 
support  plan  are  of  major  interest  during  these  activities. 

5-195.  Training  Phase 

a.  Regardless  of  the  type  of  test,  some  evaluation  of 
training  and  training  support  is  normally  conducted.  This  is 
necessary  to  ensure  the  skills  and  knowledge  necessary  to  operate 
and  maintain  a  system  can  be  attained  and  sustained  within 
realistic  training  environments  by  units  using  personnel  of  the 
type  and  qualification  expected  when  the  system  is  deployed. 

When  training  is  an  issue,  MANPRINT  and  training  data  collection 
must  begin  prior  to  T-date  (at  the  start  of  player  training) . 

b.  Conducting  new  equipment  training  (NET)  is  the  MATDEV's 
responsibility.  NET  transfers  knowledge  gained  during  materiel 
development  to  trainers,  users,  and  support  personnel  during 
development  and  fielding  of  new  equipment.  The  contents  of  the 
NET  SP  are  described  in  Part  One. 
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c.  TRADOC  is  responsible  for  the  analysis,  design, 
development,  implementation,  and  control  of  resident  training 
programs  and  exportable  training  products.  The  TRADOC  school 
responsible  for  the  MOS  affected  by  the  test  item  will  prepare  a 
training  TSP  with  contents  as  described  in  Part  One. 

d.  The  extent  of  training  and  training  support  evaluations  is 
contingent  on  the  test  type  and  stage  of  development  of  the 
system  being  tested.  Ordinarily,  training  is 
contractor-administered  in  the  early  phases  of  materiel 
development.  For  subsequent  phases,  the  MATDEV  provides  training 
to  military  instructor  personnel,  who  then  train  test 
participants.  The  objective,  however,  remains  the  same:  to 
assess  the  adequacy  of  training  associated  with  fielding  the 
system. 


e.  Test  officers  are  responsible  for  ensuring  that  test 
directorate  and  player  personnel  are  adequately  trained.  This 
often  requires  coordination  with  support  divisions  and  TRADOC 
proponent  schools.  It  is  also  important  to  ensure  that  test 
player  personnel  satisfy  test  requirements  in  terms  of  MOS  and 
skill  level.  Training  includes  that  necessary  for  controllers, 
support  personnel,  data  collectors,  and  data  reducers. 

f.  Training  conducted  in  support  of  tests  will  include 
training  individuals,  crews,  and  units  in  individual  and 
collective  tasks  required  to  employ  the  system  in  accordance  with 
approved  doctrine  and  tactics.  Training  will  be  in  accordance 
with  the  training  SP  and  representative  of  that  intended  to 
support  the  system  when  initially  fielded.  The  proponent  TRADOC 
school  must  provide  the  test  organization  and  Headquarters, 

TEXCOM  with  certification  stating  test  players  have  been  trained 
and  can  perform  individual  and  collective  tasks  to'  standard  in 
accordance  with  the  milestone  schedule  in  the  OTP.  This  written 
statement  constitutes  one  element  of  the  OTRS  but  is  provided 
separately  from  other  elements  of  the  training  developer's  OTRS. 

g.  All  training  provided  to  player  personnel,  any  performance 
problems  during  the  test  attributable  to  inadequate  training,  and 
comments  of  personnel  who  received  the  training  must  be  recorded 
and  subsequently  analyzed. 

h.  Data  are  collected  during  the  training  phase  if  required 
by  the  TEP,  If  the  TEP  does  not  require  training  phase  data,  the 
test  officer  may  wish  to  collect  these  data  as  a  training  device 
for  his  data  management  personnel  and  as  an  opportunity  to 
perform  an  end-to-end  data  run. 

5-196.  Test  Support 
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Adequate  support  is  essential  to  any  test  execution.  The  test 
officer  must  ensure  that  all  logistical  and  administrative 
requirements  that  are  planned  or  becomes  necessary  for  the  test 
execution  are  properly  performed.  The  requirements  and  plans  for 
support  are  documented  in  the  test  support  plan  for  the  test. 

5-197.  Pilot  Test  Phase 

a.  A  pilot  test  is  an  abbreviated  version  of  the  actual  test 
and  is  conducted  in  advance  to  detect  deficiencies  in  the  plan, 
instrumentation,  data  collection,  data  management  and  test 
control.  It  includes  the  exercise  of  each  type  of  required  event 
and  makes  use  of  each  data  collection  means.  It  is  essential 
that  the  complete  data  management  procedure,  to  include  DAG 
operational  procedures,  be  verified  as  a  part  of  the  pilot  test. 

b.  The  pilot  test  is  provided  for  in  the  test  design  plan 
with  sufficient  time  between  pilot  test  and  the  start  date  of  the 
actual  test  to  allow  for  identification  of,  reaction  to,  and 
correction  of  any  deficiencies  encountered.  Tests  relying 
heavily  on  instrumentation  may  require  additional  time  after  the 
pilot  test  for  the  correction  of  problems.  Accomplishment  of  an 
abbreviated  program  of  events  is  usually  sufficient,  although  an 
abbreviated  control  procedure  may  also  be  required. 

c.  If  a  pilot  test  is  not  required,  it  is  to  be  explicitly 
stated  in  the  TEP.  When  extensive  training  of  player  personnel 
is  required,  a  pilot  test  may  he  conducted  concurrently  with  the 
training  test  phase. 

d.  Problems  revealed  during  the  pilot  test  are  to  be 
corrected  prior  to  the  actual  test.  This  may  involve  the  conduct 
of  additional  training,  identification  of  additional  support, 
resources,  changes  to  the  DTP,  or  revision  to  the  TEP. 

e.  The  length  of  the  pilot  test  must  permit  the  exercise  of 
every  type  of  major  event  required  in  the  test,  as  well  as  every 
type  of  data  collection  instrument  to  be  used.  There  should  be 
enough  workdays  between  the  end  of  the  pilot  test  and  actual 
T-date  to  incorporate  any  necessary  changes. 

f.  Test  directorate  organizations  must  duplicate  those 
envisioned  for  the  actual  test  and  all  directorate  members  must 
participate.  The  degree  of  player  participation  must  be  tempered 
by  considering  if  learning  during  the  pilot  test  would  bias 
results  of  the  actual  test. 

g.  Data  should  be  collected  and  reduced  in  the  same  manner  by 
the  same  personnel  to  be  used  during  the  actual  test.  A  complete 
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end-to*end  data  run  must  be  conducted.  This  starts  with  test 
events  and  goes  through  every  step  until  a  data  base  is  created. 

h.  All  manual  data  collection  forms  must  be  validated  and  all 
instrumentation,  from  stopwatches  to  computers,  used.  The  need 
for  filming  test  events  should  be  carefully  reviewed.  Video  tape 
is  an  excellent  way  to  record  data;  however,  the  data  reduction 
and  analysis  effort  associated  with  this  medium  can  be  lengthy 
and  tedious. 

i.  If  the  test  involves  a  two-shift  operation,  data  review 
procedures  must  be  established  and  validated  during  the  pilot 
test. 


(1)  One  of  the  best  methods  of  injecting  quality  control 
into  the  data  collection  effort  is  for  the  data  manager  or 
assistant  data  manager  to  be  present  at  the  shift  change  to 
review  collected  information.  All  data  collection  forms  must  be 
complete;  for  example,  all  blocks  filled  in,  narrative  comments 
legible,  dated,  and  signed. 

(2)  Incomplete  forms  indicate  the  data  collector  does  not 
understand  his  job  or  he  is  not  interested  in  doing  his  job 
right.  In  either  case,  the  problem  must  be  resolved  prior  to  the 
test  commencement. 

(3)  Requiring  individuals  who  have  just  worked  a  12-hour 
shift  to  remain  on  duty  until  their  forms  are  corrected  is  a  good 
way  to  communicate  the  idea  that  the  data  are  important  and 
anything  less  than  perfection  is  unacceptable. 

(4)  The  conduct  of  data  reviews  and  debriefings  at  shift 
changes  is  essential. 

j.  Upon  completion  of  the  pilot  test,  all  test  directorate 
personnel  should  be  critiqued  on  their  performance  and  encouraged 
to  surface  questions  and  discuss  problems  they  encountered.  It 
is  essential  for  all  test  directorate  personnel  to  understand 
their  responsibilities  and  to  know  who  to  contact  should  a 
problem  occur. 

k.  Ensure  adequate  provisions  exist  for  such  things  as  mess, 
billets,  transportation,  communication,  drinking  water,  and 
coffee.  Adjustments  may  be  required  to  correct  deficiencies 
revealed.  This  may  involve  conducting  additional  training, 
requesting  additional  support,  revising  control  procedures, 
altering  the  test  directorate  organization,  and  revising  data 
collection  forms. 
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l.  All  problems  surfaced  during  the  pilot  test  must  be 
addressed.  They  will  not  go  away  during  actual  testing.  All 
issues  will  be  discussed  and  resolved  at  OTRR  3,  the  last  OTRR. 
It  will  be  chaired  by  the  Test  Director.  This  review  will  give 
the  go-ahead  to  start  the  test. 

m.  Contingent  upon  the  desires  of  the  test  proponent, 
customer,  or  approval  authority,  data  collected  during  training 
and  the  pilot  test  may  or  may  not  be  considered  valid.  This  is 
particularly  true  for  RAM  data.  Use  of  these  data  should  be  in 
accordance  with  the  approved  TEP  and  associated  FD/SC.  These 
data  must  be  comparable  and  compatible  with  the  data  from  record 
trials.  If  any  of  the  data  from  the  pilot  test  is  used  as  data 
in  the  test  report,  it  must  be  obtained  under  the  same  test 
conditions  as  the  record  trials. 


Section  XV 
Test  Conduct 


5-198.  Test  Structure 

The  heart  of  test  is  typically  divided  into  sub-tests  or  phases 
which  are  controlled  by  operational  scenarios  consisting  of 
trials  and  events.  Tests  may  have  as  many  phases  as  necessary. 
Typically,  an  FTX  maneuver  will  suffice.  Phases  may  be  conducted 
at  separate  locations,  separate  test  facilities,  or  at  one 
location  or  facility.  A  class  of  sub-tests  is  special  exercises 
and  excursions.  These  sub-tests  are  typically  small-scale  series 
of  events,  or  special  circumstances  used  to  explore  special  cases 
or  particular  situations  which  do  not  occur  naturally  in  test. 
They  can  originate  based  on  unexpected  results  from  current  test, 
questions  raised  by  senior  management,  or  attempts  to  answer 
"what  if"  questions. 

5-199.  Adherence  to  Test  Design 

After  completing  all  necessary  adjustments  to  the  TEP  and  DTP, 
the  test  officer  must  ensure  that  the  test  is  conducted  according 
to  the  updated  plans.  Test  change  proposals  (TCP)  must  be 
approved  by  the  same  headquarters  that  approved  the  TEP  (see 
Chapter  3).  Major  deviations  from  the  test  plan,  such  as  those 
which  require  changes  in  test  milestones  or  which  affect  the 
potential  accomplishment  of  one  or  more  sub-objectives,  are  to  be 
immediately  brought  to  the  attention  of  the  test  proponent  and 
other  appropriate  commands  and  agencies.  All  deviations  from  the 
approved  TEP  are  immediately  and  fully  documented  and  reported. 
When  retesting  or  necessary  extensions  to  the  test  impact  on 
program  schedules,  the  CBTDEV,  MATDEV,  and  the  operational  tester 
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are  to  recommend  alternative  test  programs  and  risks  to  ASA(ROA) 
or  other  appropriate  approval  authority. 

5-200.  Test  Changes 

During  the  test,  it  may  become  apparent  that  deviation  from  the 
TEP  or  DTP  is  required.  Minor  adjustments  to  these  plans  are 
frequently  required  during  test  execution,  and  are  made  by  the 
test  officer.  The  test  officer  should  consult  with  the  analyst 
assigned  to  support  his  test  to  determine  the  impact  that  his 
changes  may  have  on  the  statistical  validity  of  the  test.  The 
test  officer  must  also  ensure  that  he  adheres  to  the  policies 
established  within  his  test  activity  for  implementing  minor 
adjustments  to  the  test  plan.  Major  deviations  from  the  test 
plan,  such  as  those  which  require  changes  in  test  milestones  or 
which  impact  on  answering  one  or  more  issues,  should  immediately 
be  brought  to  the  attention  of  appropriate  personnel.  Any 
changes  to  the  approved  TEP  must  be  approved  by  the  headquarters 
which  approved  the  TEP  through  submission  of  a  TCP. 

5-201.  Checklist 

During  the  various  phases  of  a  test,  the  test  officer  has  many 
actions  that  need  to  be  completed.  To  help  ensure  that  these 
actions  are  performed  on  time,  a  test  execution  checklist  has 
been  developed  and  is  at  figure  5-11. 

(INSERT  FIGURE  5-11) 

5-202.  Test  Officer  Log 

The  test  officer  log  is  a  notebook,  preferably  with  numbered 
pages,  that  enables  the  test  officer  or  representative  to  record 
indelibly  (no  erasures)  all  the  major  events  of  the  test,  , 
beginning  at  the  start  of  the  pilot  test.  A  chronological  order 
with  events  listed  in  military  time  is  the  most  convenient 
format.  They  include,  but  are  not  limited  to,  weather 
conditions,  personnel  injuries,  equipment  or  instrumentation 
conditions,  absences  of  test  or  player  personnel,  threat 
simulator  status,  test  anomalies,  adequacy  of  collected  data, 
security  and  safety  matters,  mission  start  and  stop  times, 
frequency  authorization  and  usage,  reasons  for  any  delays,  and 
any  incident  requested  by  the  data  analysts  to  include  in  the  TR 
time  event  charts.  The  test  officer  log  is  the  official  on-site 
record  of  test  events.  The  exact  format  should  be  determined 
prior  to  T-date.  The  log  should  be  complete  and  faithfully 
maintained. 

5-203.  Test  Schedule 

Once  the  test  begins,  the  test  officer  is  responsible  for  seeing 
that  the  approved  plan  is  followed.  This  requires  substantial 
supervision  and  control  of  all  resources  dedicated  to  the  test. 
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He  must  ensure  that  the  prescribed  scenario  is  followed,  that 
support  activities  are  functioning,  and  that  data  are  collected 
and  stored  properly.  OPSEC  of  data  and  equipment  is  a  major 
concern.  Equipment  left  at  the  test  site  overnight  must  be 
protected  from  damage  by  the  elements  and  from  theft  or 
vandalism.  The  test  officer  must  also  ensure  that  the  safety 
plan  is  followed.  Duties  such  as  presenting  briefings  should  be 
delegated  to  , assistant  test  officers. 

5-204.  Quick  Look  Reports 

Information  from  the  test  officer  log  and  other  sources  can  be^ 
summarized  in  the  format  of  a  cursory  "quick  look"  report.  This 
information  may  be  disseminated  in  accordance  with  test 
directorate  standard  operating  procedures  or  prior  agreements. 

The  test  officer  is  responsible  for  the  contents  and  accuracy  of 
this  report.  It  should  be  limited  to  one  or  two  pages  and 
provide  basic  information  for  a  specific  area.  No  quick  look 
report  should  be  disseminated  outside  the  test  directorate 
without  written  release  by  the  director  or  other  appropriate 
authority. 

5-205.  Visitors  and  Briefings  _  ,  , 

The  test  officer  must  be  prepared  to  thoroughly  brief  visitors  on 
the  purpose,  scope,  and  conduct  of  the  test.  Resources  necessary 
to  do  this  must  be  planned  and  included  in  the  OTP.  Resources 
may  include  facilities,  transportation,  quarters,  and  dedicated 
briefing  officer.  Coordination  will  be  required  with  the  local 
installation  protocol  officer  for  VIP  visitors.  The  test  officer 
can  control  a  large  number  of  visitors  by  scheduling  a  visitor's 
day  just  prior  to  start  of  the  pilot  test.  In  this  manner,  a 
large  number  of  PMs,  CBTDEV,  MATDEV,  unit  chain  of  command,  and 
other  visitors  will  be  accommodated  without  impact  on  record 
trials.  Visitors  day  also  serves  as  an  internal  check  to  ensure 
everything  is  ready  for  T-date.  During  the  conduct  of  record 
trials,  it  is  inevitable  that  visitors  will  arrive  and  a  thorough 
plan  with  an  updated  briefing  is  required  to  answer  their 
questions  and  minimize  interference  with  record  trials. 

5-206.  External  Personnel 

During  testing,  there  may  be  numerous  personnel  on  site.  Some 
will  stay  for  the  duration  of  the  test.  Personnel  from  PM,  TSM, 
CBTDEV,  MATDEV,  TNGDEV,  and  DT  agencies  are  vitally  interested  in 
OT.  The  test  officer's  primary  mission  is  the  conduct  of  the 
test.  To  do  this  more  efficiently,  the  test  officer  should 
establish  early  and  firmly  plans  to  accommodate  personnel  not 
assigned  directly  to  the  test  team. 

5-207.  Test  Incident  Reporting 
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During  test  execution,  each  intermittent  or  incipient 
malfunction,  safety  hazard,  or  degradation  in  system  performance 
or  operation  is  to  be  documented  in  a  test  incident  report  (TIR) 
prepared  by  the  organization  conducting  the  test.  TIRs  provide  a 
standard  method  of  reporting  incidents  disclosed  during  materiel 
testing.  Copies  of  TIRs  describing  significant  major  acquisition 
program  T&E  events  as  defined  in  the  TEMP  (such  as  missile 
launches  or  live  firings)  are  provided  to  the  DOTE  and  DDDRE(T&E) 
within  24  hours  after  the  event  (AR  73-1) . 

5-208.  Postponement  of  Testing 

a.  In  the  event  that  an  OTRR  indicates  that  testing  should  be 
delayed  (i.e.,  inadequacies  of  SSPs,  OTRS,  training,  test 
planning,  instrumentation,  completeness,  safety  or  health 
hazards) ,  alternative  courses  of  action  and  recommendations  are 
developed  which,  if  executed,  assist  in  maintaining  the  integrity 
of  the  test.  Due  to  resource  provider  1-year  prior  notification 
requirements  for  UTs,  seemingly  short  delay  requirement  could 
result  in  a  delay  of  1  year  or  more  (See  AR  15-38) . 

b.  Scheduled  testing  may  be  postponed  upon  identification  of 
a  major  problem  (i.e.,  those  problems  which  would  impact  the 
validity  of  the  data  being  collected  to  address  critical  issues) 
during  testing  or,  when  it  is  apparent  that  the  system 
performance  has  little  chance  of  successfully  attaining 
operational  criteria.  User  Testing  may  be  delayed  by  the  user 
tester. 


c.  Initial  approval  to  postpone  a  test  must  be  obtained  from 
within  the  test  and  evaluation  organization.  The  proponent  or 
evaluator,  if  not  previously  informed,  and  the  materiel  developer 
(if  the  test  was  for  a  materiel  system)  will  be  notified  of  the 
suspension  within  24  hours.  Notification  will  include  the 
rationale  for  the  action.  Resources  will  not  be  released  without 
the  permission  of  the  headquarters  which  approved  the  TEP. 

d.  Upon  suspension,  the  program  sponsor  will  convene  a 
program  review  to  consider  future  actions.  Once  the  program 
sponsor  has  a  solution  to  the  problem,  the  TIWG  will  be  convened 
to  determine  necessary  additional  tests  to  validate  the  fixes. 
Before  restart  of  testing,  appropriate  test  readiness  reviews 
will  be  conducted  with  the  same  level  of  participation  as  in 
previous  OTRRs. 

e.  Only  the  headquarters  which  approved  the  TEP  may  terminate 
a  test  prior  to  completion.  Termination  involves  the  release  of 
resources. 
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Section  XVI 

Test  Data  Management 


5-209.  Overview  ^  ^ 

The  following  sections  provide  details  of  test  data  management 
processes,  the  levels  of  test  data  and  their  relation  to  the  test 
report,  and  presents  some  suggested  procedures  for  use  during  the 
data  collection,  reduction,  and  analysis  processes. 


5-210.  Purpose  . 

The  purpose  of  data  management  is  the  collection,  quality 
control,  reduction,  and  storage  of  test  data  generated  during 
test  execution  into  a  more  condensed  and  usable  format  for 
analysis  and,  ultimately,  into  test  results  and  evaluation. 

The  process  of  data  collection  and  reduction  should  involve 
collecting  and  processing  data  during  the  pilot  test  as  well  as 
the  record  test.  Obviously,  not  all  the  data  are  available 
during  early  test  execution,  but  the  test  of ficer /analyst  must 
start  the  data  reduction  phase  as  soon  as  possible  in  order  to 
reveal  unusable  data  and  meet  test  reporting  milestones.  No 
system  should  proceed  into  record  test  until  end-to-end  data  runs 
have  been  successfully  performed  in  the  pilot  test  for  each 
element  of  the  test  data  base.  This  will  be  a  prime  agenda  item 
at  the  OTRR  just  prior  to  start  of  test. 

5-212.  Data  Tracking  and  Audit  Trail  , 

The  OT  results  have  a  great  potential  to  influence  the  overall 
decision-making  process;  therefore,  great  care  must  be  exercised 
to  ensure  that  they  are  properly  formulated  to  accurately  depict 
the  results  of  system  performance  during  the  test.  The  method 
and  rationale  used  to  collect,  record,  and  process  the  test  data 
and  to  derive  test  results  should  be  recorded  to  maintain  an 
audit  trail  and  facilitate  report  writing. 


5-213.  Data  Management  Requirements  o  •  4. 

Requirements  for  test  data  management  are  depicted  in  the  Data 
Management  Plan  (see  above) .  The  data  management  requirements 
describe  the  data  management  organization  and  procedures  used  to 
ensure  that  the  test  data  base  accurately  reflects  what  occurs 
during  the  test.  Data  processing  are  the  procedures  through 
which  recorded  test  data  is  organized,  reduced,  verified, 
managed,  controlled,  and  stored.  Data  management  personnel 
organize  the  data  in  terms  of  the  collection  source  (RAM 
collection  form,  cockpit  digital  recorder,  radar  tapes,  etc.)  and 
perform  the  planned  procedures  through  which  each  set  of 
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collected  data  pass  before  reaching  the  storage  medium  which 
supports  the  test  directorate  and  the  evaluator. 

5-214.  Data  Processing 

Data  processing  is  described  in  the  ASP  (see  above) .  Data 
processing  begins  when  completed,  properly  formatted  data  are 
received  from  the  data  collection  section.  It  ends  when  the 
final  test  data  base  is  produced.  This  includes: 

a.  Entering  manual  and  automated  data  into  an  automated  data 
base. 

b.  Performing  both  manual  and  automated  reduction. 

c.  Execution  of  quality  control  procedures. 

d.  Constructing  and  maintaining  files  to  provide  a  complete 
test  data  audit  trail  from  collection  (instrumentation  tapes, 
videotapes,  and  paper  files  of  original  manual  data  forms,  data 
collector  logs,  controller  logs.  Interviews,  etc.)  through  any 
interim  reduction  process  to  the  final  test  data  base. 

5-215.  Data  Flow 

The  data  flow,  both  within  the  data  management  section  and 
between  elements  of  the  test  directorate  must  be  developed  and 
documented  for  use.  It  documents  procedures  and  schedules  to  be 
used  for  data  transmittal  especially  manual  transmittal 
operations.  Automated  formats  are  described  in  detail  in  the 
ASP.  The  data  flow  includes  procedures  where  data  is  combined 
with  other  data,  where  it  is  processed,  scored,  reorganized, 
validated,  or  otherwise  manipulated.  This  includes  transfer  of 
data  from  data  forms  to  automated  form.  Particular  attention  is 
paid  to  feedback  loops,  both  within  the  data  management  section 
and  among  data  control,  collection,  and  management  elements.  In 
addition,  feedback  loops  from  emerging  evaluation  and  statistical 
analyses  are  generally  established. 

5-216.  Organization  for  Data  Management 

a.  The  organization  of  the  data  management  section,  including 
functions  and  responsibilities  of  key  section  personnel  and 
personnel  rotation  schedules  must  be  established  and  provided  to 
all  personnel.  Responsibilities  of  key  personnel  and  overall 
organizational  structure  are  designed  to  accommodate  data  entry, 
data  reduction  and  validation,  and  audit  trail  preservation. 

b.  The  data  management  section  typically  requires  personnel 
from  several  agencies  or  contracting  organizations  at  many 
different  levels  possessing  skills  from  a  wide  variety  of 
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disciplines.  Organization  of  the  section  should  allow  for 
unusual  duty  schedules  or  freguent  rotations  of  technical 
personnel  to  permit  timely  reduction  of  test  data  and  to 
accommodate  personnel  constraints  of  other  agencies. 

c.  Training  for  data  management  personnel  on  the  flow  of  data 
and  procedures  to  be  used  within  the  section  and  on 
responsibilities  of  section  personnel  must  be  planned  and 
conducted.  Training  is  to  be  tailored  to  meet  individual  needs. 
Duties  of  personnel  within  the  section  tend  to  be  specialized, 
ranging  from  entering  and  filing  data  to  reviewing  reliability 
and  maintainability  descriptions. 

5-217.  Data  Requirements  Support 

Instrumentation,  AV,  and  ADP  must  be  incorporated  into  data 
management  planning  for  OTE.  This  planning  results  in 
development  of  three  specific  documents  which  are  incorporated 
into  the  DTP:  the  ISP,  the  AVSP,  and  the  ASP.  Each  of  these 
plans  were  described  previously  in  this  chapter.  Each  of  these 
plans  provide  the  details  of  support  to  the  data  management 
processes  as  described  below. 

a.  Instrumentation.  Scientific  or  technical  equipment,  to 
include  threat  simulators  and  targets,  used  to  measure,  sense, 
record,  transmit,  process,  or  display  data  during  tests, 
examination  of  materiel,  training  concepts  or  tactical  doctrine. 

b.  Audiovisual  (AV) .  Audio  and  visual  documentary  coverage 
of  how  testing  was  conducted,  as  well  as  collection  of  selected 
data  elements  for  incorporation  into  the  report.  The  AV 
documentation  covers  the  testing  and  other  tester  activities  to 
include  test  unit,  test  directorate,  side  tests/demonstrations, 
instrumentation,  data  collection,  and  data  processing. 

c.  Automated  Data  Processing  (ADP) .  Input,  storage, 
computing,  control,  and  output  services  of  test  data,  using 
electronic  circuitry  within  a  computer  system  to  perform 
arithmetic  and  logical  operations  by  means  of  internally  stored 
or  externally  controlled  programmed  instructions 


Section  XVII 

Test  Data  Collection 


5-218.  General  .  ^ 

The  data  collection  process  includes  the  organization  and 
procedures  which  will  ensure  that  all  required  data  are  collected 
correctly,  verified,  and  properly  formatted  for  storage. 
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retrieval,  and  analysis.  Data  are  usually  collected  in  three 
different  ways. 

a.  Automated  instrumentation  with  associated  data  bus  and 
data  distribution  system,  including  telemetry  links  recorded  on 
hard  disks  or  magnetic  tape. 

b.  Manually  by  data  collectors  with  data  collection  sheets 
for  specific  portions  of  the  test  such  as  RAM,  MANPRINT,  or 
notebooks,  including  the  TO  Log  that  are  annotated  by  key  test  or 
player  personnel. 

c.  Optically  or  acoustically  with  video  cameras,  microphones, 
or  other  transducers  that  have  a  self-contained  recording  system. 

5-219.  Organization  for  Data  Collection 

a.  The  organization  and  procedures  for  data  collection  (and 
quality  control  of  the  collection  process)  must  be  established  to 
ensure  that  data  collectors  and  instrumentation  will  be  in  the 
right  places  at  the  right  time.  It  must  describe  when,  where, 
and  how  checks  of  data  forms  or  instrumentation  outputs  will  be 
made  to  ensure  that  data  are  being  collected  and  recorded 
according  to  the  definitions  and  formats  required. 

b.  The  data  collection  section  typically  consists  of  teams 
which  are  assigned  to  data  collection  stations.  These  stations 
may  be  moving  (e.g.,  with  a  tank  in  the  attack),  permanent  (e.g., 
at  the  same  location  for  the  whole  test),  or  temporary  (e.g.,  at 
a  given  location  during  only  one  part  of  the  test). 

c.  Whereas  controllers  are  concerned  with  the  test  input  or 
stimulus,  data  collectors  are  concerned  with  the  test  conditions 
and  the  output  or  response  of  the  test  item  and  player 
participants.  Some  data  collectors  may  be  tasked  to  operate 
automatic  or  semiautomatic  data  collection  equipment. 

d.  As  a  minimum,  procedures  must  provide  for  the  following 
items: 


(1)  Composition.  A  chain  of  responsibility  is  established 
and  mission  statements  are  provided  as  required. 

(2)  Allocation  of  collection  means.  Each  data  recording 
agent  (human  or  instrument)  is  assigned  to  a  player  unit  or 
location,  and  the  data  to  be  collected  are  identified.  These 
arrangements  are  closely  dependent  upon  the  control  plan. 
Provisions  are  made  for  such  items  as  relief  of  data  collectors 
and  instrumentation  battery  changes. 
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(3)  Gathering.  Procedures  and  schedules  are  established 
for  the  periodic  pick  up  and  control  of  manually  recorded  data 
from  collectors.  Prompt  consolidation  of  data  reduces  the 
likelihood  of  loss  and  allows  concurrent  monitoring  for 
incompleteness  or  error. 

(4)  Data  recovery.  Procedures  are  specified  for  attempting 
recovery  of  data  which  are  lost  or  erroneously  not  collected, 
such  procedures  are  to  be  instituted  quickly  while  the  data  may 
still  be  available  or  can  most  easily  be  reconstructed. 

(5)  Coordination.  Plans  are  made  for  continuous  monitoring 

of  the  test  events.  Changes  in  the  sequence  or  of  events 

may  require  reallocation  of  data  collection  ®eans. 
accomplished  by  coordination  between  the  control  headquarters  and 
the  data  collection  headquarters. 

5-220.  Data  Collector  Training 

a.  The  training  plan  roust  include  training  to  be  given  to 
data  collectors  and  instrumentation  operators  as  well  as 
procedures  to  determine  whether  training  was  effective.  Data 
collectors  require  familiarity  with  the  control  plan  and  the  data 
management  and  collection  plans,  as  well  as  the 

scope,  subtests  and  field  exercises,  test  item  characteristics, 
instrumentation,  duty  schedules,  and  communications  procedures 
and  radio  nets. 

b.  Detailed  training  in  the  correct  use  of  data  collection 
forms  is  always  required.  Particular  emphasis  is  given  to  the 
need  for  accuracy  and  unbiased  objectivity.  Selected  data 
collectors  and  supervisors  may  require  detailed  training  on 
operations  or  maintenance  of  the  tested  system. 

c.  Training  is  to  stress  that  data  collectors  are  not  to  cue 
test  players  to  pending  events.  Data  collectors  should  also  be 
trained  to  implement  emergency  procedures  if  necessary. 

5-221.  Data  Collection  Forms 

a.  Data  which  is  manually  collected  will  require  data 
collection  forms.  Proper  data  form  design  considers  the  data 
audit  trail  (in  particular,  providing  space  for  identifying  data 
cSliLto?  U  quality  control  reviewers)  and  the  need  to  minimize 
or  preclude  entry  errors  or  omissions.  Form  design  guidelines 
include: 
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(1)  A  field  requiring  a  certain  number  of  characters  or 
digits  should  have  blocks  or  underlines  which  clearly  indicate 
field  length. 

(2)  Special  formats  should  be  indicated  (e.g.,  MM:SS  for  a 
time  in  minutes  and  seconds) . 

(3)  Multiple  choice  questions  should  generally  not  require 
data  collectors  to  remember  input  codes  but  should  either  define 
the  codes  or  provide  for  check-off  blanks. 

(4)  Input  of  extraneous  data  should  be  avoided  (e.g., 
requiring  entry  of  year  or  even  month  for  a  test  conducted 
10-25  June  1988) . 

b.  Some  redundancy  may  be  desirable  to  ensure  the 
traceability  of  forms.  For  instance,  a  mission  number  could  be 
paired  with  other  records  to  identify  system,  date,  location, 
etc. ,  but  a  transposed  or  missing  digit  in  either  number  could 
render  a  form  entirely  untraceable. 

c.  Data  collection  form  instruction  sheets  are  developed 
which  describe  when  and  how  each  entry  on  each  data  collection 
form  will  be  obtained  and  entered.  The  definitions  in  the  data 
element  dictionary  in  the  TEP  and  the  data  reduction  plan  are 
considered  when  preparing  these  instructions. 

5-222.  Data  Collection  and  Instrumentation 
Requirements  for  collection  of  data  by  instrumentation  are  a 
major  part  of  many  tests.  Procedures  must  be  developed  which 
describe  the  requirements  for  instrumentation  equipment  and 
personnel  (operators  and  maintainers)  to  ensure  that  adequate 
physical  facilities  are  available  and  that  appropriate  personnel 
training  is  accomplished.  It  also  describes  how  the  complete 
instrumentation  system  will  be  tested  and  verified  prior  to,  or 
as  part  of,  the  pilot  test.  This  may  include,  in  conjunction 
with  other  collection  means',  the  planning  and  performance  of  an 
end-to-end  data  processing  trial. 

5-223.  Logistics  Requirements 

The  data  collection  cell  identifies  peculiar  logistics  support 
requirements  to  the  test  directorate  describing  any  particular 
logistics  requirements  of  the  data  collection  section,  especially 
transportation,  communications,  and  rations. 


Section  XVIII 
Test  Data  Reduction 
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5-224.  General  .  ..  . 

The  data  reduction  process  consists  of  data  processing  through 
level  4.  When  practical,  this  process  begins  during  the  test 
execution  phase  (or  as  soon  as  raw  data  becomes  available) , 
rather  than  waiting  until  the  test  execution  phase  has  been 
completed.  This  is  necessary  to  ensure  the  highest  degree  of 
quality  control  at  the  earliest  time  when  the  data  is  fresh. 


5-225.  Data'  Reduction  Plan 

It  is  the  responsibility  of  the  test  officer  to  ensure  that  da 
reduction  is  adequate  to  provide  the  data  in  a  form  that  the 
analysts  can  use  productively.  A  coherent  data  reduction  plan 
showing  how  the  data  will  be  processed,  transformed,  and/or 
presented  is  vital  to  the  test  program.  The  plan  should  explain 
how  the  data  can  be  scrutinized  quickly  for  the  post-mission 
briefings  and  how  the  data  will  be  handled  for  more  thorough 
analysis. 


5-226.  Data  Reduction  Methods 

Data  can  be  reduced  in  several  ways,  ranging  from  central 
processing  with  a  high  capacity  computer,  to  manual  extraction 

and  correlation. 


a.  Generally,  a  central  processor  is  required  for  automated 
instrumentation  data  such  as  that  for  position  location.  This 
data  is  read  from  hard  disk  or  magnetic  tape  and  presented  on  a 
high  speed  printer  or  displayed  graphically  from  a  CRT  or 
plotter . 


b.  Manually  collected  data  is  usually  compiled  and  sorted  in 
matrix  form.  It  can  be  entered  by  keyboard  on  a  central 
processor  for  future  integration  into  other  data  bases  or  entered 
on  a  personal  computer  for  storage  and/or  limited  manipulation. 


c  Video  and  voice  data,  collected  on  tape,  is  replayed  on  a 
monitor  with  IRIG  time  or  other  key  parameters  superimposed  on 
the  screen.  The  analyst  can  reduce  this  data  manually  but  many 
of  the  recorded  parameters  can  be  entered  by  keyboard  on  the 
central  processor  to  be  reformatted  and  become  part  of  an  overall 

data  base. 


d.  Additional  common  techniques  may  be  employed,  but  the 
important  goal  for  the  test  officer  is  to  recover  the  reduced 
data  fast  enough  to  enable  the  analyst  to  confirm  its  validity 
before  too  many  missions  have  been  completed. 


e  Interim  reduction  consists  of  manual  or  automated 
procedures  which  transform  data  from  the  data  collection  means  to 
the  form  required  in  the  final  data  base. 
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f.  Manual  reduction  includes  processes  requiring  expert 
analyses  and  judgment  such  as  extracting  quantitative  data  from 
videotapes  and  examining  data  from  multiple  sources  to  resolve 
apparent  contradictions. 

g.  When  extensive  manual  reduction  and  validation  is  needed, 
a  DAG  may  provide  limited  support.  However,  DAG  autonomy  and 
independence  must  be  preserved  by  ensuring  that  the  DAG  does  not 
participate  in  any  of  the  required  steps  in  the  data  reduction 
process. 

h.  Automation  is  used  whenever  it  can  sensibly  replace  manual 
reduction  procedures,  and  such  automation  support  is  documented 
in  the  automation  support  plan. 

5-227.  Data  Reduction  Review 

At  the  conclusion  of  the  data  reduction  sub-phase,  in-house 
personnel  should  review  the  data  and  the  planned  methodology  for 
data  analysis.  Level  1  through  4  data  should  be  available  for 
this  review  to  enable  tracking  of  the  data  reduction  process.  At 
each  step  that  data  changes  to  a  different  level,  quality  control 
is  critical.  The  newly  ordered  data  must  accurately  and 
faithfully  represent  the  test  events  that  triggered  it.  Quality 
checks  must  be  exercised  during  pilot  test  and  periodically 
during  record  trials. 

5-228.  Data  Turn-around 

Data  must  be  turned  around  as  expeditiously  as  possible.  Data 
entry  and  data  reduction  should  not  tolerate  any  backlogs.  If 
data  collection  forms  backup,  quality  control  is  difficult, 
because  the  trail  is  cold  when  errors  are  discovered.  Expedited 
production  of  test  data  assists  MATDEV  pursuit  of  appropriate 
corrective  actions,  permits  the  test  officer  and  evaluator  to 
begin  analysis  and  report  generation,  and  allows  time  for 
discovery  and  investigation  of  anomalies  while  the  test  is  still 
in  progress.  Slow  turn-around  of  data  reflects  poorly  on  the 
competence  of  the  test  organization  and  the  test  director. 


Section  XIX 

Test  Data  Quality  Control 


5-229.  General 

Quality  control  is  the  process  for  independently  validating  test 
data.  It  defines  the  checks  and  procedures  which  are  to  be  used 
to  preclude  or  detect  and  correct  errors  made  in  data  collection, 
data  entry,  or  data  reduction.  It  also  identifies  emerging  data 
summaries  required  to  identify  potential  inconsistencies.  It 
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outlines  the  process  for  making  required  corrections  or  changes 
in  test  data  and  how  an  audit  trail  for  those  corrections  or 
changes  is  to  be  maintained. 

5-230.  Data  Verification 

Data  verification  determines  whether  the  data  correctly  represent 
the  variables  they  characterize,  and  whether  the  data  collected 
are  adequate  to  support  the  OTE  reports. ^  Specify  requirements 
for  and  techniques  proposed  for  data  verification.  Data 
collection  procedures  are  to  be  validated  prior  to  starting  the 
test  and  checked  during  the  test  to  ensure  that  critical  data  are 
being  accurately  collected.  Expedited  data  turn-around 
procedures  are  necessary  to  assure  that  quality  control  checks 
are  timely.  This  permits  reconstruction  of  erroneous, 
conflicting,  confused,  or  incomplete  data  while  collectors  and 
participants  memories  are  still  fresh. 

5-231.  Quality  Control  Procedures 

The  data  management  officer  must  develop  in  detail  the  data 
management  procedures  which  ensure  that  the  correct  data  are 
collected  and  that  the  data  collected  are  correctly  portrayed. 
Data  management  quality  control  procedures  provide  mechanisms  for 
adding  or  changing  data  requirements  and  for  modifying  data 
collection  or  reduction  procedures  based  on  examination  of 
emerging  test  data.  This  differs  from  the  quality  control 
procedures  in  data  collection  which  are  generally  limited  to 
assuring  that  data  are  completely  and  accurately  collected. 

5-232.  Quality  Control  Checks 

Both  manual  and  automated  checks  are  used  to  ensure  that  data  are 
correctly  entered  and  portray  test  events  accurately .  Although 
manual  checks  often  include  extensive  review  of  computer 
printouts,  they  should  not  generally  involve  extensive  arithmetic 
operations.  Such  operations,  together  with  repetitive  logical 
checks,  should  be  automated  to  the  maximum  extent  and  documented 
in  the  data  reduction  plan.  Manual  checks  consist  primarily  of 
checks  which  cannot  be  easily  automated  (such  as  review  of 
graphical  displays)  and  assessment  of  related  data  from  several 
sources  for  overall  consistency. 

5-233.  Quality  Control  Limitations 

The  data  management  responsibility  is  to  ensure  that  data  taken 
as  a  whole  represent  what  actually  happened  during  the  test.  It 
stops  short  of  evaluative  analyses  of  test  data,  which  are 
primarily  concerned  with  the  interpretation  and  significance  of 
test  data  within  the  broader  framework  of  a  system  evaluation. 

5-234.  Trial  Validation 
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^  The  validation  process  describes  the  procedures  (and 
organization  if  appropriate)  for  determining  the  acceptability  of 
trial  conduct  and  validity  of  trial  results  prior  to  data 
reduction.  The  process  of  validating  trials  shortly  after  their 
conduct  leads  to  timely  recognition  of  invalid  trials,  timely 
rescheduling  of  trials  which  fail  to  meet  minimum  standards  and 
efficiency  in  data  reduction  because  invalid  data  is  segregated 
before  processing. 

b.  Decisions  on  trial  validity  are  to  be  made  by  the  test 
officer  with  input  from  blue  and  red  forces  command, 
instrumentation,  and  test  control  spokespersons.  DAG  personnel 
may  be  involved  in  the  validation  process  as  appropriate  (eg, 
provide  technical  expertise,  etc.),  but  may  not  participate  or 
vote  in  it. 


Section  XX 
OTE  Data  Bases 


5-235.  Data  base  design 

The  data  base  design  is  the  process  whereby  the  structure  and 
content  of  the  data  base  for  storage  of  the  test  data  is 
formulated,  created,  and  documented.  The  architecture  and  design 
of  the  data  base  are  described  including  the  relationships  among 
files  and  records.  The  data  in  each  file  or  record  are  to  be 
listed  and  augmented  by  any  necessary  definitions.  Good  data 
definitions  specify  exactly  what  is  measured  when.  Examples  are; 

a.  "Elapsed  time  to  transmit  a  call  is  to  be  measured  and 
recorded  to  the  nearest  second  by  a  data  collector  at  the 
transmitter  using  a  stopwatch.  Transmission  time  begins  when  the 
operator  .  .  .  <specify  operator  action>  and  ends  when  .  .  . 
<specify  operation>." 

b.  "RMS  display  range  is  the  RMS  range  between  the  aircraft 
and  the  ground  target  at  the  time  when  the  1553  data  bus  in  the 
aircraft  confirms  that  the  ground  target  symbol  is  displayed  on 
the  aircraft  gunner's  scope.  This  is  available  from  the  RMS 
instrumentation  system." 

5-236.  Data  Base  Preparation 

When  test  data  are  extensive  enough  to  require  storage  in  an 
automated  data  base,  the  structure  and  content  of  the  data  base 
are  prepared  in  accordance  with  the  architecture  and  design  of 
the  data  base  established  during  test  planning.  The  data 
management  section  ensures  that  test  data  is  entered  into  the 
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data  base  in  accordance  with  the  preplanned  relationships  among 
files  and  records. 


5-237.  Management  of  the  Data  Base  , 

Data  base  management  describes  the  procedures  to  ensure  integrity 
of  the  test  data  base.  In  particular,  it  specifies  data  entry 
and  updating  procedures  which  preclude  transcription  errors  and 
unauthorized  access  and  provides  for  a  complete  audit  trail  for 
the  data. 


5-238.  Data  Base  Report  . 

The  data  base  structure  and  data  dictionary  (listing  and 
description  of  all  codes  used  in  the  data  base)  become  the  core 
of  the  Data  Base  Description  (Appendix  A  of  the  TER  or  TR) . 


Section  XXI 
DAG  Operations 


5-239.  DAG  Purpose  . 

The  DAG  authenticates  and  validates  the  test  data,  ensuring  that 
test  data  accurately  reflect  the  system  performance  during  the 
test  and  provide  the  single  test  database  of  record  (the  ground 
truth)  for  all  users  of  the  test  data.  The  DAG  identifies  and 
analyzes  anomalies  in  the  system  under  test,  instrumentation,  and 
test  data.  The  DAG  provides  interested  agencies  a  conduit  to 
express  opinions  during  test  planning  and  execution. 

5-240.  Establishment  of  the  DAG 

The  lOE  establishes  the  requirements  for  a  DAG  on  full-evaluate 
system  tests.  These  requirements  are  documented  In  the  TEP.  If 
the  lOE  does  not  require  a  DAG,  the  tester  determine  if  a  need 
exists  and  establishes  a  DAG.  The  tester  determines  if  a  need 
exists  and  establishes  a  DAG  for  an  abbreviated  evaluate  systems 
and  for  FDTE,  CEP,  and  CT. 

5-241.  DAG  Roles  and  Missions 

a.  The  DAG  serves  the  function  of  bringing  together  the 
interested  parties  on  an  operational  test  and  allowing  these 
parties  to  view  test  planning,  execution,  and  data  reduction. 

DAG  members  provide  recommendations  to  the  evaluator  and  tester 
on  matters  of  test  design,  test  conduct,  and  test  data  reduction. 
It  provides  a  level  of  quality  assurance  above  that  expected  from 
the  data  management  quality  control  function.  The  DAG  acts  as 
advisory  group  to  the  Test  Director  and  the  Chief  Evaluator. 
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b.  Due  to  the  variations  in  development  systems,  evaluation 
strategies,  test  designs,  and  data  collection  efforts,  the  duties 
of  each  DAG  are  specifically  tailored  to  accommodate  the  unique 
requirements  of  the  test.  The  evaluator  and  the  tester  carefully 
define  the  relationship  between  the  DAG  and  the  other  elements  of 
the  test  directorate. 

c.  The  DAG  acts  independently  of  the  data  management  and 
quality  control  process  and  does  not  work  under  the  supervision 
of  the  data  manager.  It  provides  a  level  of  quality  assurance 
above  that  to  be  expected  from  the  data  management  quality 
control  function. 

d.  DAG  members  will  review  and  authenticate  the  test  conduct, 
data  collection,  data  reduction  and  data  base  contents  as 
indicated  by  the  SOP.  The  DAG  will  identify  and  investigate  any 
problems,  discrepancies,  or  anomalies  found  in  these  areas,  and 
make  recommendations  to  the  Test  Director  for  resolution  of  these 
problems.  The  DAG  verifies  that  the  data  contained  in 
performance,  human  factors,  and  RAM  test  data  bases  are  valid 
test  results.  The  DAG  will  publish  reports  as  required.  The  DAG 
serves  to  promote  T&E  community  understanding  and  acceptance  of 
the  operational  test  data. 

e.  Final  decisions  on  test  design,  test  conduct,  and  test 
data  reduction,  lie  solely  with  the  tester  and  evaluator. 

5-242.  DAG  Membership 

a.  The  lOE  chairs  the  DAG  for  full-evaluate  system  tests. 

The  tester  chairs  the  DAG  (when  established)  for  all  other  tests. 

b.  The  DAG  charter  establishes  DAG  membership.  Mandatory 
members  are  the  evaluator  and  tester.  Select  other  members  from 
the  Combat  Developer,  Materiel  Developer,  Technical  Evaluator, 
Technical  Tester,  and  other  members  of  the  acquisition  team. 
Extend  membership  to  any  pertinent  government  agency  (DOTE,  AAA, 
GAO)  with  a  vested  interest  in  the  system  under  test.  The 
members  of  the  DAG  represent  a  broad  spectrum  of  technical 
disciplines  and  system  expertise. 

c.  Each  DAG  is  organized  to  accommodate  the  unique 
requirements  of  the  test.  Large  DAGs  are  typically  organized 
into  various  functional  teams  such  as  a  performance  validation 
team,  a  MANPRINT  data  validation  team,  a  RAM  data  validation 
team,  and  a  research  cell.  Small  DAGs  may  consist  of  one  cell. 

d.  The  law  (10  USC  2366)  prohibits  direct  participation  in 
the  DAG  by  system  contractors.  The  DAG  permits  no  system 
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contractor  manipulation  or  influence  during  lOT  and  other 
activities  which  provide  input  for  consideration  during  beyond 
LRIP  decisions  for  ACAT  I  and  II  systems.  System  contractor 
personnel  will  not  attend  or  be  directly  involved  as  members  or 
observers  in  any  DAG  sessions. 

e.  Support  contractors  to  DAG  members  may  participate  in  the 
DAG  if  they  have  never  had  a  contractual  relationship  to  the 
system  contractor  on  the  system  under  test. 

5-243.  DAG  Resources 

All  resources  for  the  functions  of  the  DAG  must  be  included  in 
the  OTP  for  the  test.  Resources  for  personnel,  travel, 
equipment,  facilities,  and  overtime,  must  be  estimated  by  the 
tester,  with  input  from  the  lOE. 

5-244.  DAG  Training 

The  DAG  cannot  function  properly  if  the  members  do  not  have 
adequate  training.  Training  should  be  addressed  in  the  DAG  SOP 
and,  as  a  minimum,  members  should  have  training  in  operations  and 
capabilities  of  the  system  under  test,  familiarization  with  test 
purpose  and  concept  as  documented  in  the  TEP  and  DTP ,  the  data 
reduction  plan  and  instrumentation  for  the  test,  the  DAG  SOP,  and 
test  organization  and  key  personnel. 

5-245.  Data  Levels 

a.  The  originator  of  the  requirement  for  the  DAG  determines 
the  data  levels  to  be  reviewed  by  the  DAG  and  addresses  this  in 
the  DAG  SOP.  Each  member  of  the  DAG  should  be  clear  on  the 
meanings  of  each  data  level  as  given  in  Figure  5-12. 

(INSERT  FIGURE  5-12) 

b.  The  DAG  SOP  may  call  for  examination  of  data  from  levels 
1-3  in  the  authentication  process.  Once  the  level  3  data  base 
has  been  reviewed  and  approved  by  the  DAG,  it  becomes  the 
authenticated  data  base,  which  is  the  database  of  record  for  that 
test.  Authenticated  data  are  releasable  to  members  of  the 
acquisition  team. 

c.  The  analysts  can  reduce  and  analyze  these  data  into 
findings  and  assessments  (levels  4,  5,  6,  and  7).  The  levels  of 
data  and  their  relationship  to  the  reduction  and  analysis  process 
and  the  categories  of  test  results  are  illustrated  in  Figure  5- 
12. 
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Test  Data  Analysis  Process 


5-246.  General 

Data  analysis  consists  of  performing  those  analytical  techniques 
defined  in  the  TEP  and  further  amplified  in  data  reduction  and 
analysis  planning.  Normally  this  phase  involves  the  use  of 
inferential  statistics  and  the  application  of  logic,  common 
sense,  and  military  judgment  to  data  of  levels  3  through  6  for 
the  purpose  of  identifying  findings  and  making  assessments  and 
evaluations.  ' 

5-247.  Evaluation  Plan  or  Analysis  Plan 

a.  Essentially,  the  correct  analytical  approach  starts  with 
the  evaluation/analysis  dendritic,  the  MOE/MOP  list,  and  the  Data 
Requirements  list.  The  lists  correlate  (through  the  dendritic) 
to  each  criterion  and,  thence,  to  each  issue. 

b.  The  Evaluation  Plan  (Analysis)  plan  in  Chapter  2  of  the 
TEP  states  how  the  collected  and  reduced  data  relating  to  each 
MOE/MOP  will  be  used  to  answer  the  respective  criteria  and  how 
all  the  answered  criteria  will  address  the  issues. 

c.  The  plan  also  addresses  the  sample  size  and  confidence 
level  for  any  measured  parameters  (e.g.,  operational  availability 
(Ap)  for  any  hypothesis  testing  required  as  part  of  a  MOP.  The 
adjustments  to  sample  sizes  are  a  vital  part  of  the  test  program 
because  they  directly  affect  cost  and  resources. 

d.  The  plan  is  dynamic  in  the  respect  that  it  may  require 
continuous  updating  during  test  execution  as  a  function  of  any 
unforeseen  events  that  affect  the  validity  of  the  data  base.  The 
tester,  evaluator,  and  analysts  must  periodically  review  the 
adequacy  of  the  plan  to  ensure  that  it  will  properly  answer  all 
of  the  questions  for  each  issue  and  criteria. 

e.  Review  and  analysis  of  the  data  will  often  lead  analysts 
to  investigate  analytic  procedures  not  envisioned  in  the  plan. 
Emerging  results  may  often  suggest  new  approaches.  The  tester 
and  evaluator  have  the  latitude  to  perform  these  investigations 
so  long  as  they  are  documented  in  the  report. 

5-248.  Data  Analysis  Techniques/Procedures 

Data  analysis  techniques  or  procedures  formulate  a  subject  of 
great  extent  and  complexity.  They  range  from  the  logical 
qualitative  approach,  such  as  orderly  arrangement  of  test 
incident  reports  (Terse) ,  to  curve  fitting  of  probability  density 
and  distribution  functions  by  regression  methods.  Many  of  these 
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techniques  are  usually  an  analytical  or  engineering  specialty, 
but  the  evaluator,  tester,  and  analysts  should  have  a  basic 
understanding  of  how  they  are  used.  Choose  analytical  personnel 
for  the  OT&E  team  whose  background  is  compatible  with  the  level 
of  difficulty  involved  in  required  data  analyses. 

5-249.  Analytic  Review  Board  ^ 

^fter  the  data  has  been  analyzed  and  findings,  assessments,  and 
accompanying  discussion  have  been  formulated,  an  analytic  review 
board  may  be  held  to  review  results  prior  to  writing  the  report. 
The  board  would  give  appropriate  personnel  an  opportunity  to 
review  the  analytic  work  and  question  the  logic  or  techniques 
which  resulted  in  findings  and  assessments.  Format  for  this 
review  should  be  the  question  and  answer  type,  and  require  no 
formal  presentation. 

5-250.  Data  Base  Formats 

As  the  manipulation  of  test  data  becomes  more  and  more 
sophisticated,  it  is  essential  that  some  type  of  data  base  format 
be  established.  However,  because  of  the  diversity  of  the  data 
and  the  various  techniques  that  can  be  used  to  manipulate  the 
data,  it  is  difficult  to  use  only  one  format  for  all  data 
reduction.  Use  a  common  data  software  packages  for  analysis 
(i.e.,  SAS,  DBase  III  or  IV,  Lotus  1-2-3).  Unless  otherwise 
approved,  use  one  of  these  formats  when  formatting  a  data  base. 
Data  displays  will  be  as  specified  in  the  TEP. 


Section  XXIII 
Test  Data  Reporting 

5-251.  General  .  ^  ^  ^ ^ 

This  process  identifies  data  display  requirements  and  anticipated 
displays  (both  management  displays  and  report  displays) . 
Management  displays  summarize  the  status  of  the  test  or  test^ 
events.  Report  displays  present  (or  in  special  cases  summarize) 
test  data.  Display  requirements  specify  how  the  data  are  to  be 
organized,  formatted,  and  displayed. 

5-252.  Test  Management  Displays 

Management  display  procedures  describe  any  data  listings,  tabular 
summaries,  and  graphical  displays  to  be  used  to  monitor  and 
control  test  conduct,  and  assure  data  quality.  Two  examples  of 
such  displays  are:  tables  of  conditions  associated  with  test 
events  conducted  (for  comparison  with  controller  records)  and 
lists  of  sorted  data  (to  identify  possibly  anomalous  data  for 
detailed  review) . 

5-253.  Report  Displays 
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a.  Report  displays  (tester  and  evaluator)  present  (or  in 
special  cases  summarize)  test  data.  Specifications  for  each 
report  include  the  data  to  be  displayed,  the  factors  and 
conditions  by  which  they  are  to  be  organized,  the  selection 
criteria  for  determining  whether  a  data  point  is  to  be  included, 
and  guidance  as  to  the  form  of  the  display. 

b.  Summary  statistics  or  graphs  are  used  in  test  report 
displays  only  for  large  sets  of  data  (more  than  can  reasonably  be 
displayed  on  one  page)  and  in  these  cases,  selection  criteria  and 
reference  to  the  data  base  are  to  be  identified. 

c.  Data  presentations  are  not  to  include  any  judgments  or 
evaluative  treatment  by  the  tester.  Judgmental  information,  such 
as  test  directorate,  user,  or  expert  observer  opinions,  are  to  be 
explicitly  identified  as  such  and  presented  as  derived  in  a 
separate  display.  The  tester  does  not  decide  among  disagreements 
or  take  responsibility  for  the  correctness  of  judgmental  data. 

d.  Good  data  displays  do  not  lead  the  reader  to  any 
conclusions  concerning  the  presented  test  data.  For  this  reason, 
the  best  test  report  displays  are  tabular. 

5-254.  Procedures  for  Release  of  Data 

a.  The  test  directorate,  in  accordance  with  established 
policy,  describes  procedures  for  access  to  data  and  for  release 
of  interim  data  outside  the  test  and  evaluation  teams.  Access  to 
test  data  is  generally  granted  lAW  AR  25-55,  The  Department  of 
Army  Freedom  of  Information  Act  Program,  10  January  1990. 

b.  All  data  is  releasable  to  the  tester  and  the  evaluator. 
Authenticated  test  data  may  be  released  to  the  acquisition  team 
members,  HQDA  staff,  and  to  DOTE  based  on  need-to-know  and 
security  considerations.  However,  release  of  interim  data  is 
generally  discouraged. 

c.  Any  release  (including  reports  from  test  and  evaluation 
team  members  to  superiors  outside  the  test  directorate)  is  to  be 
dated  and  is  to  clearly  indicate  the  preliminary  nature  of  the 
data. 


SECTION  XXIV 
Completion  of  Test 


5-255.  Official  Completion  Date 
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The  test  is  officially  completed  at  the  scheduled  completion  date 
unless  unforeseen  difficulties  have  precluded  collection  of  all 
required,  analyzable  data.  Proper  conduct  of  the  test  will 
enable  the  test  officer  to  make  the  final  decision  regarding 
acceptable  test  data.  If  the  collected  data  in  the  final 
missions  of  the  test  program  are  not  acceptable,  the  procedure 
for  extending  the  test  as  discussed  in  ?????????  will  be 
followed. 

5-256.  Post  Test  Activities 

a.  The  test  officer  must  remain  on  the  test  site  or  revisit 
the  site  as  necessary  to  phase  out  all  operations.  The  equipment 
under  test  and  all  other  external  resources  must  be  returned  to 
their  normal  places.  The  test  area  should  be  policed  to  clear 
out  litter  and  other  expendable  items.  Care  must  be  taken  to 
ensure  that  personnel,  anxious  to  return  to  their  normal  duty 
stations,  do  not  allow  lapses  in  security  to  occur  or  to  neglect 
necessary  close-out  duties. 

b.  All  test  and  player  personnel  must  acknowledge  completion 
of  their  assigned  duties  before  release.  A  list  of  all  these 
persons  and  their  addresses  should  be  retained  to  provide 
appropriate  written  recognition  of  their  efforts.  Any  test  event 
causing  a  potential  adverse  effect  on  the  environment  should  be 
reported  to  senior  directorate  personnel  and  appropriate  post 
agencies . 


SECTION  XXV 

Special  Considerations 

5-257.  Contractor  Relations 

a.  The  intent  of  Section  910,  Subsection  2366,  paragraph 
(b) (2)  of  Public  Law  99-661  is  to  ensure  that,  during  OT,  major 
weapon  systems  are  operated,  maintained,  and  otherwise  supported 
by  personnel  typical  of  those  who  will  carry  out  such  duties  when 
the  system  is  deployed  in  combat.  Therefore,  this  paragraph  will 
be  applied  to  the  test  and  evaluation  of  ACAT  I  programs  and  is 
recommended  for  application  to  the  test  and  evaluation  of  all 
programs  (See  AR  73-1) . 

b.  To  ensure  there  is  no  system  contractor  manipulation 
and/or  influence  during  lOTE  or  activities  which  provide  input 
for  consideration  in  the  operational  evaluation  leading  to  a  full 
production  decision  for  ACAT  I  programs: 

(1)  System  contractor  personnel  will  not: 
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(a)  Participate  except  to  the  extent  they  are  involved 
in  the  operation,  maintenance,  and  other  support  of  the  system 
when  it  is  deployed. 

(b)  Participate  in  collection,  reduction,  processing, 
authentication,  scoring,  assessing,  analysis,  or  evaluation  of 
operational  test  data. 

(c)  Attend  or  be  directly  involved  as  members  or 
observers  in  OT  DAG  sessions  (see  above  in  this  chapter)  or  in 
RAM  scoring  or  assessment  conferences  (see  Part  One)  which 
address  test  data  supporting  operational  evaluation  or  assessment 
of  their  systems. 

(2)  Discussions  with  system  contractor  personnel  may  be 
necessary  to  ensure  full  technical  understanding  of  observed  test 
incidents  during  the  above  lOTE  or  activities.  Each  and  every 
such  discussion  will  be  held  separate  from  any  scoring/assessment 
activities.  A  written  record  of  the  nature  of  these  contractor 
to  government  discussions  will  be  maintained  by  the  test  officer 
or ' independent  evaluator,  as  appropriate. 

c.  Since  some  systems  will  be  maintained  by  contractors  after 
fielding,  it  is  imperative  that  any  contractor  effort  be  defined 
in  writing  prior  to  T-date.  Ideally,  any  authorized  contractor 
maintenance  would  be  specified  by  level  and  extent  in  each  of  the 
appropriate  test  support  packages.  Contractor  efforts  should  be 
an  agenda  item  briefed  at  the  T-60  OTRR,  and  agreed  to  by  all 
parties.  EUTE  and  FDTE  prior  to  lOTE  will  often  require  a 
greater  amount  of  contractor  maintenance  support,  but  this  must 
be  worked  out  in  the  TIWG. 

d.  Although  the  above  legal  strictures  apply  only  to  ACAT  I 
systems  during  lOTE  or  any  FOTE  leading  to  a  beyond  LRIP,  full 
production  decision,  it  is  DA  policy  that  these  strictures  should 
apply  to  all  OT  of  materiel  and  IMA  systems. 

5-258.  Release  of  Test  Information 

a.  Release  of  OT  data  to  members  of  the  acquisition  team 
(MATDEV,  CBTDEV,  TNGDEV,  development  and  operational  evaluators, 
and  development  and  operational  testers  is  authorized  as  soon  as 
the  data  are  authenticated  (Level  3  data) .  Release  is  also 
authorized  to  TEMA,  ODUSA(OR) ,  and  DOTE.  The  operational  tester 
is  authorized  to  release  these  data. 

b.  Release  of  OT  data  beyond  the  acquisition  team  will  be 
accomplished  only  with  the  approval  of  CG,  OPTEC  or  the  commander 
of  other  OTE  activities.  All  such  requests  for  data  should  go  to 
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the  evaluator  who  will  then  coordinate  with  the  tester  and  the 
PEO  or  PM. 

c.  The  conduct  of  operational  tests  on  new  materiel  has 
gained  widespread  interest,  resulting  in  numerous  requests  for 
interim  OTE  data.  These  requests  are  generated  by  congressional 
survey  and  investigatory  committees,  the  General  Accounting 
Office  (GAO),  the  Army  Audit  Agency  (AAA),  industry,  contractors, 
and  private  individuals. 

d.  Any  requests  for  test  information  received  by  the  test 
team  from  members  of  news  medie  or  civic  organizations  should  be 
reported  immediately  to  the  appropriate  agency  public  affairs 
officer. 

e.  The  test  officer's  primary  mission  is  the  conduct  of  the 
test.  All  requests  from  outside  the  acquisition  team  for  interim 
data  or  reports  should  first  be  coordinated  with  the  program 
manager  (PM)  or  program  executive  officer  (PEO) .  Requests  for 
information  from  private  industry  or  individuals  will  be 
processed  as  public  information  releases  or  Freedom  of 
Information  Act  (FOIA)  requests.  Directives  addressing  the 
release  of  information  must  be  used  for  guidance.  (See  AR  1-20, 
AR  25-55  and  AR  360-5.) 

f .  Release  of  draft  or  interim  test  reports,  evaluations,  or 
assessments  is  discouraged.  Assessments  made  prior  to  the 
complete  analysis  of  test  results  can  be  very  misleading,  can  be 
found  to  be  incorrect  when  the  complete  set  of  test  data  is 
thoroughly  analyzed,  and  can  cause  biases  which  are  difficult  to 
overcome  even  when  further  information  proves  them  incorrect. 

g.  Release  of  interim  data  or  reports  will  not  be  made 
without: 

(1)  Verification  of  the  requester's  identity  and  need. 

(2)  Analysis  of  the  difficulty  associated  with  providing 
the  information  requested. 

(3)  Coordination  with  OPTEC  or  other  operational  test 
activity  staff. 

h.  All  data  released  will  be  as  authentic  and  complete  as 
possible.  The  data  or  report  will  be  clearly  marked  as  interim 
and  cautions  to  be  considered  in  its  utilization  will  also  be 
stated. 
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i.  Copies  of  the  release  letter  will  be  retained  in  the 
official  system  file. 

j.  Release  of  information  to  system  contractors  will  be  made 
only  through  the  PEO,  PM,  or  appropriate  materiel  developer 
representative.  Release  of  information  to  support  contractors 
will  be  made  only  through  the  COR  or  COTR. 

k.  Security  classification  and  procedures  to  protect 
classified  or  competition  sensitive  information  will  always  be 
observed . 
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Detailed  Test  Plan 
for 

(Name  of  Test) 


1.  Purpose.  (Repeat  paragraph  1.1  from  TEP.) 

2.  Scope.  (Repeat  paragraph  1.2  from  TEP.) 

3.  Background.  (Repeat  paragraph  1.3  from  TEP.) 

4.  System  description.  (Repeat  paragraph  1.4  from  TEP.) 

5.  Critical  operation  issues  and  criteria  (COIC)  or 
operational  issues  and  criteria  (OIC) .  (Repeat  paragraph  1.7 
from  TEP. ) 

6.  Test  and  evaluation  milestones.  (Repeat  paragraph  1.6 
from  the  TEP,  as  modified  by  any  subsequent  changes  since  TEP 
approval . ) 

7.  Changes  to  TEP  chapters  1,  2,  and  3. 

7.1  List  substantive  changes  to  paragraphs  contained  in 
chapters  1,  2  and  3  of  the  approved  TEP.  Changes  should  be  in 
the  form  of  replacement  paragraphs,  as  appropriate.  Each 
change  should  be  formatted  as  follows: 

”7.1  Change  to  TEP  paragraph  x.x.x: 

X. X.X  Enter  the  changed  paragraph,  to  include 
subparagraphs,  tables,  and  figures  as 
appropriate. ” 

7.2  Change  to  TEP  paragraph  y.y.y 

Y. Y.Y  Changed  paragraph.” 

8.  Summary  of  DTP  appendix  contents.  General  description  of 
the  changes,  modifications,  or  additions  to  the  approved  TEP 
that  is  contained  in  the  DTP.  The  format  should  be  general 
subparagraphs  that  summarize  the  material.  Examples  are: 

a.  Data  Management  Plan  (app  K) .  If  no  other  change  is 
necessary  this  may  simply  amount  to  adding  the  final  data 
collection  forms  for  the  test.  The  data  collection  forms  were 
not  developed  for  inclusion  in  the  TEP. 
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b.  Control  Plan.  If  this  appendix  remains  substantially 
correct  then  all  that  may  need  to  be  added  is  the  final 
organization  of  the  control  section  or  a  description  of  a  new 
radio  or  wire  net  for  test  control. 

c.  Training  Plan.  There  may  have  been  a  change  in  the 
location  of  instruction  or  instructors  which  needs  to  be 
described. 

d.  If  an  appendix  is  not  changed  from  the  corresponding 
appendix  contained  in  the  approved  TEP  and  is  not  included  in 
the  DTP,  indicate  by  adding  the  words  ”(Not  changed)" 
following  the  appendix  listing.  For  example: 

Appendix  U.  Glossary  (not  changed) . 

e.  If  an  appendix  is  simply  not  used,  so  state.  For 
example: 

Appendix  L.  Instrumentation,  Targets,  and  Threat 
Simulators  Support  Plan.  (Not  used). 

(NOTE:  Include  each  of  the  following  appendixes,  as 

appropriate,  which  are  needed  and  which  have  been  changed  from 
the  appendixes  contained  in  the  approved  TEP.  Appendix 
identification  letters  should  correspond  with  the  appendix 
letters  used  in  the  approved  TEP.  Do  not  reprint  or  include 
approved  TEP  appendixes  which  have  not  been  changed.  List  all 
appendixes  and  indicate  those  not  used  in  accordance  with  the 
note  following  the  listing.) 
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A.  Supporting  Documentation 

B.  Background — OPTIONAL 

C.  System  Description — OPTIONAL 

D.  Projected  Threat — OPTIONAL 

E.  DAG  Charter  and  SOP — Required  if  DAG  is 

called  for  in  paragraph  2.4.5.  of  the  TEP 

F.  Outline  Test  Plan  (OTP) 

G.  Pattern  of  Analysis 

H.  Test  Scenarios — OPTIONAL 

I .  Test  Threat— OPTIONAL 

J.  Control  Concept 

K.  Data  Management  Plan 

L.  ITTS  Support  Plan 

M.  Audiovisual  Support  Plan 

N.  Automation  Support  Plan 

O.  Training  Plan 

P.  Test  Support  Plan 
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APPENDIX  Q. 

Test  Environmental  Assessment 

APPENDIX  R. 

Environmental  Impact  Statement 

APPENDIX  S. 

DOT&E  Test  Concept  Approval 

APPENDIX  T 

Test  Change  Proposals 

TAB  1. 

Test  Change  Proposal  #1 

Through 

TAB  n. 

Test  Change  Proposal  #n 

APPENDIX  U. 

Glossary,  Acronyms,  and  Abbreviations 

APPENDIX  V. 

TEP  Coordination  Record 

APPENDIX  W. 

Authors  and  Supporting  Personnel 

APPENDIX  X. 

TEP  Distribution  List 
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STRAWMAN 

DATA  AUTHENTICATION  GROUP  (DAG) 

STANDARD  OPERATING  PROCEDURES  (SOP) 

1.  Purpose.  This  SOP  describes  the  DAG  operation  specific  to 
this  test.  It  includes  the  mission,  concept  of  operation, 
organization,  responsibilities,  and  expected  output. 

2.  Mission.  This  paragraph  describes  the  mission  of  the  DAG. 
It  should  be  tailored  to  the  needs  of  the  test,  and  identify 
the  actions  to  be  performed  by  the  DAG.  These  may  include 
functions  such  as  review  of  data  reduction  procedures,  review 
of  data  to  verify  that  the  contents  of  the  database  reflect 
the  events  of  the  test,  witness  of  trial  conduct  to  verify 
compliance  with  the  test  plan,  problem  investigation  and 
recommendation  of  solutions,  and  expected  DAG  output  such  as 

I  reports.  Specific  levels  of  data  to  undergo  DAG  review  should 
be  identified. 

3.  Concept  of  Operation.  This  paragraph  provides  the 
procedures  of  the  DAG  and  its  relationship  to  the  rest  of  the 
Test  Directorate.  It  should  describe  the  DAG's  interaction 
with  the  Data  Manager  at  each  level  of  data  review.  One 
approach  would  be  to  chart  the  flow  of  the  data  to  be  reviewed 
as  it  comes  from  the  data  manager  to  the  DAG  and  goes  back  to 
the  Data  Manager,  indicating  each  level  to  be  reviewed.  The 
process  for  validating  the  data  and  the  methodology  proposed 
for  tracking  data  that  has  been  validated  should  be 
documented.  This  paragraph  should  specify  the  term  of  the  DAG 
(e.g.,  T-date  through  C-date  plus  60),  the  frequency  of  data 
deliveries  to  the  DAG  for  review,  the  frequency  of  DAG 
meetings,  and  what  is  required  to  declare  the  database 
"authenticated";  i.e.,  what  will  constitute  completion  of  the 
DAG  mission.  This  paragraph  should  also  contain  the  reporting 
chain  from  the  DAG  chair  to  the  Test  Director. 

4.  Organization.  This  paragraph  provides  the  organization 
structure  and  membership  of  the  DAG  itself.  Members  may  be 
formed  into  teams  responsible  for  specific  DAG  functions. 
Special  teams  may  be  formed  as  needed  to  research  problems 
identified  by  the  DAG.  These  research  teams  may  include  non- 
DAG  members  who  provide  a  technical  expertise  needed  to 
complete  the  investigation.  System  contractor  personnel  may 
be  part  of  a  research  team,  but  may  not  attend  or  otherwise 
participate  in  the  DAG  meetings. 

_ _ _ _ _ 


Figure  5-2.  Strawman  DAG  SOP 


5.  Responsibilities.  This  paragraph  defines  the  duties  any 
DAG  member  must  perform  to  contribute  to  the  goal  of  DAG 
mission  accomplishment,  including  the  minimal  requirement  for 
regular  attendance.  It  also  should  identify  the  duties  of  the 
DAG  chair  to  organize  the  DAG  and  implement  the  SOP,  including 
the  interface  between  the  chair  and  the  rest  of  the  Test 
Directorate.  If  the  DAG  is  organized  into  teams,  the 
responsibilities  of  team  leaders  to  organize  their  teams, 
assure  completion  of  the  team  function  and  provide  regular 
status  reports  to  the  DAG  chair  should  be  listed.  The  party 
responsible  for  development  and  conduct  of  DAG  training  should 
be  identified. 

6.  Training.  This  paragraph  addresses  the  contents  and 
schedule  of  the  DAG  training  program. 

7.  Reports.  This  paragraph  addresses  the  expected  products 
of  the  DAG  including  timing,  approval  chain,  and  release 
authority. 


Figure  5-2.  Strawman  DAG  SOP  (cont) 


FORMAT  FOR  A  TEST  CONTROL  PLAN 

THE  TEST  CONTROL  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <yYYY>  SYSTEM 

1.  CONTROL  CONCEPT. 

1.1.  Purpose. 

1.2.  Scope . 

1.3.  Approach . 

1.4.  Level  of  Operational  Realism. 

1.5.  Synopsis  of  Events. 

1.6.  Control  Methods. 

2.  PROGRAM  OF  EVENTS. 

2.1.  Test  Outline. 

2.2.  Detailed  Test  Schedule. 

2.3.  Overall  Test  Scenario. 

2.4.  Detailed  Test  Scenario. 

2.5.  Event  Summary. 

2.6.  Scenario  Revision  and  Documentation. 

2.7.  Pilot  Test 

3 .  CONTROL  PROCEDURES . 

4.  DOCUMENTATION. 

4.1.  Controller  Logs. 

4.2.  Test  Status  Reporting. 

4.3.  Historical  Documents. 

4.4.  Submission  Schedules. 

5.  CASUALTY  ASSESSMENT. 

6.  CONTROL  ORGANIZATION. 

6.1.  Organization  Chart. 

6.2.  Control  Section. 

6.3.  Control  Teams 

6.4.  Communications  Control 

6.5.  Test  Operations  Center 


Figure  5-3.  Format  for  a  Test  Control  Plan 
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FORMAT  FOR  A  DATA  MANAGMENT  PLAN 

THE  DATA  MANAGMENT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <YYyY> 
SYSTEM 

1.  Data  Management  Concept. 

1.1.  Organization. 

1.2.  Data  Flows . 

1.3.  Methods  of  Operation. 

2.  Data  Collection  Procedures. 

2.1.  Collection  Methods. 

2.1.1.  Manual  Recording  by  Players. 

2.1.2.  Manual  Recording  by  Data  Collectors. 

2.1.3.  Automated  Collection  by  Instrumentation 

2.1.4.  Questionnaires. 

2.1.5.  Data  Collection  Forms. 

2.1.6.  Structured  Interviews. 

2.2.  Collection  Support. 

2.2.1.  Initial  QC  Procedures. 

2.2.2.  Debriefing  Procedures. 

2.2.3.  Instrumentation. 

2.2.4.  Data  Collectors  with  Special  Qualifications. 

2.2.5.  Data  Security  Requirements 

2.2.6.  Special  Data  Handling  Requirements. 

2.2.7.  Video  and  Photography  Documentation. 

3.  Data  Reduction  Procedures. 

3.1.  Data  Entry 

3.1.1.  Manual. 

3.1.2.  Automated. 

3.2.  Data  Processing. 

3.3.  Data  Verification/Validation. 

3.4.  Data  Storage. 

3.5.  Data  Manipulation. 

3.6.  Data  Presentation. 

4.  Data  Base  Design. 

4.1.  Identification  of  the  Required  Files. 

4.2.  Data  Base  Architecture. 

4.3.  Definition  of  Data  Elements. 

4.4.  Data  Inputs  and  Outputs 

a.  Data  name 

b.  System  name 

c.  Width  (char) 

d.  Format 

e.  Edit  checks 

f.  Source 


Figure  5-4.  Format  for  a  Data  Management  Plan 
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FORMAT  FOR  A  DATA  MANAGMENT  PLAN 

THE  DATA  MANAGMENT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <yYYY> 
SYSTEM 

5.  Data  Base  Structure. 

5.1.  Data  Element  Dictionary. 

a.  Data  name 

b.  System  name 

c.  Data  description 

d.  Range  of  the  data 

e.  Data  source 

5.2.  Sample  Forms. 

5.3.  Data  Base  Management. 

6.  Quality  Control  (QC) . 

6.1.  Manual  QC  Checks. 

6.2.  Automated  QC  Checks. 

6.3.  Data  summaries. 

6.4.  Audit  Trails. 

7 .  Output  Reports . 

7.1.  Test  Report . 

7.2.  Data  Displays. 

7.3.  DAG  Reports . 

7.4.  Quality  Control  Reports. 

7.5.  Scoring  Conference  Reports. 

7.6.  Test  Incident  Reports. 


Figure  5-4.  Format  for  a  Data  Management  Plan  (cont) 
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FOMAT  FOR  AN  ITTS  SUPPORT  PLAN  (ISP) 

ITTS  SUPPORT  PLAN  FOR  THE  <XXXX>  TEST  FOR  THE  <YyYY>  SYSTEM 

1.  ITTS  Support  Concept. 

1.1.  Purpose . 

1.2.  Scope . 

1.3.  Approach . 

1.4.  Brief  System  Description. 

2.  Instrumented  Data  Collection. 

2.1.  Data  Requirements . 

2.2.  Collection  Considerations. 

2.2.1.  Goal . 

2.2.2.  Redundancy . 

2.2.3.  Movement . 

2.2.4.  Additional  Equipment. 

3.  Interface  with  ADP. 

4.  ITTS  Requirements. 

4.1.  Nummber  and  Type  of  Evaluated  Systems. 

4.2.  Data  Inputs. 

4.3.  Mobility. 

4.4.  Scenarios. 

4.5.  Security. 

4.6.  Power  Sources. 

4.7.  Collection  Locations. 

4.8.  Soldier  Monitoring . 

4.9.  Candidate  Systems. 

5.  Other  Support  Requirements. 

6.  Test  Support. 

6.1.  Milestones. 

6.2.  Demonstrations. 

6.3.  Training. 

6.4.  Pilot  Test. 

6.5.  OTRR. 

7.  Conduct  of  Test. 

8.  Special  Instrumentation  Considerations. 


Figure  5-5.  Format  for  an  ISP 
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FORMAT  FOR  AN  AUDIOVISUAL  SUPPORT  PLAN  (AVSP) 

THE  AUDIOVISUAL  SUPPORT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <YYYY> 
SYSTEM 

1.  AUDIOVISUAL  CONCEPT. 

1.1.  Purpose. 

1.2.  Scope . 

1.3.  Approach . 

1.4.  Support  Description. 

1.4.1.  Significant  Events. 

1.4. 1^1.  Detectability. 

1.4. 1.2.  Tactical  Nobility. 

1 . 4 . 1 . 3 .  Engagement  Sequences . 

1.4. 1.4.  MANPRINT. 

1.4. 1.5.  Still  Layouts. 

1.4. 1.6.  Targets. 

1.4.2.  Personnel  and  Training. 

1.4.3.  Procedures . 

2.  OPERATIONAL  REQUIREMENTS. 

2.1.  Equipment. 

2.1.1.  Video. 

2.1.2.  Still. 

2.2.  Subject  Areas  for  Support. 

2.2.1.  <name>  (1st  Subject  Area  for  Support). 

2 . 2 . 1 . 1 .  Type  of  support  required . 

2. 2. 1.2.  Quantity. 

2. 2. 1.3.  Location. 

2. 2. 1.4.  Plan. 

2.2.2.  <name>  (2nd  Subject  Area  for  Support). 

(through) 

2.2. n.  <name>  (n-th  Subject  Area  for  Support). 

3.  SCHEDULING. 

3.1.  Pretest. 

3.2.  Pilot  Test. 

3.3.  Conduct  of  Test. 

4.  SPECIAL  REQUIREMENTS  AND  CONTINUOUS  COVERAGE. 

4.1.  Non-Standard  Tests., 

4.2.  Limited  Resources. 

4.3.  Support  Requirements . 

4.4.  Contingencies. 

4.4.1.  Equipment  Failure. 

4.4.2.  Changing  Requirements. 


Figure  5-6.  Format  for  an  AVSP 
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FORMAT  FOR  AN  AUTOMATION  SUPPORT  PLAN  (ASP) 

THE  AUTOMATION  SUPPORT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <yYYY> 
SYSTEM 

1.  ADP  Support  Concept. 

1.1.  Organization. 

1.2.  Data  Flow  and  Anticipated  Volume  of  Data. 

1.3.  Method  of  Operation. 

1.3.1.  ,  Equipment  or  Services. 

1.3.2.  System  or  Software. 

2.  Design  Details. 

2.1.  ADP  Functional  Description. 

2.2.  Detailed  Characteristics. 

2.2.1.  Performance  Requirements. 

2. 2. 1.1.  Accuracy. 

2. 2. 1.2.  Timing. 

2. 2. 1.3.  Inputs  and  Outputs. 

a.  Data  name. 

b.  System  name. 

c.  Width  (char). 

d.  Format. 

e.  Edit  checks. 

f.  Source. 

2.2.2.  Failure  Contingencies. 

2. 2. 2.1.  Backup. 

2. 2. 2. 2.  Fallback. 

2.3.  Data  Entry. 

2.3.1.  Manual. 

2.3.2.  Automated. 

2.4.  Data  Processing. 

2.5.  Data  Verification/Validation. 

2.6.  Quality  Control. 

2.7.  Storage. 

2.8.  Data  Manipulation. 

2.9.  Output  Reports . 

2.9.1.  Test  Report . 

2.9.2.  Data  Displays. 

2.9.3.  Other  Preliminary  Reports. 

2.10.  Analysis  Requirements. 

2.11.  External  Interfaces. 

3.  Data  Base  Structure. 

3.1.  Dictionary. 

3.2.  Sample  Forms. 

3.3.  Output  Reports . 

Figure  5-7.  Format  for  an  ASP 


FORMAT  FOR  AN  AUTOMATION  SUPPORT  PLAN  (ASP) 

THE  AUTOMATION  SUPPORT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <YYYY> 
SYSTEM 

4.  Environment. 

4.1.  Hardware. 

4.2.  Communications. 

4.3.  Facilities. 

4.4.  Cost . 

5.  System  Development. 

5.1.  Software  Development. 

5.2.  User's  Manual. 

5.3.  Coordination. 

5.4.  Training. 

6 .  Test  Support . 

6.1.  Milestones. 

6.2.  Demonstrations. 

6.3.  Training. 

6.4.  Pilot  Test. 

6.5.  OTRR. 

6.6.  Control  and  Retention  of  Data  Base. 


Figure  5-7.  Format  for  an  ASP  (cont) 
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FORMAT  FOR  AN  TEST  SUPPORT  PLAN  (TSP) 

THE  TEST  SUPPORT  PLAN  FOR  THE  <XXXX>  TEST  OF  THE  <YYYY>  SYSTEM 

1.  Test  Support  Concept. 

1.1.  Purpose. 

1.2.  Scope . 

1.3.  Approach  to  Test  Support . 

1.4.  Organization  for  Test  Support 

2.  Logistics  Support. 

2.1.  Supply. 

2.2.  Maintnenace. 

2.3.  Transportation. 

2.4.  Facilities. 

2.5.  Services. 

2.6.  Medical. 

3.  Administrative  Support. 

3.1.  Communications 

3.1.1.  Radio . 

3.1.2.  Wire. 

3.2.  Detailed  Cost  Estimate. 

3.3.  Safety. 

3.4.  Electronic  Warfare. 

3.5.  Visitor  Control. 

3.6.  Public  Affairs. 

3.7.  OPSEC. 

3.8.  Security. 

4.  Long  Lead  Items. 

4.1.  Ammunition. 

4.2.  Aircraft. 

4.3.  Instrumentation. 

4.4.  Personnel  Skills. 

4.5.  Equipment  Fabrication. 

4.6.  Contracts. 

4.7.  Environment. 

4.8.  Target  Alteration. 

4.9.  Permanent  Construction. 

4.10.  Electrical  Power. 

4.11.  Liquid  Petroleum  Gas/Water/Septic  Tanks. 

4.12.  Nonstandard  Stock. 

4.13.  Rental  Car  Support. 

4.14.  Telephone/FAX  Lines/Radios  (Communications). 

4.15.  Computer  Support . 

5.  Test  Support  Schedule. 


Figure  5-9.  Format  for  an  TSP 
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OTRR  AGENDA 


1.  Purpose 

2.  Program  Sponsor  Issues  (Program  Sponsor) 

a.  Results  of  Previous  Testing 

b.  System  Equipment  Status 

c.  Operational  Test  Readiness  statement 

d.  Safety  Release 

e.  System  Delivery  Schedules  (Milestone) 

f.  Contractors  Support 

g.  Logistics  Support  Plan 

h.  Instrumentation 

i.  System  Transfer  Plan 

j.  Certification  of  Systems  Readiness  for  OT 

k.  Other  Special  Topics 

3.  Combat  Developer /Trainer  Issues  (Combat 
Developer /Trainer ) 

a.  Test  Soldier  Training  Results 

b.  Operational  Test  Readiness  Statement 

c.  Safety  Release 

d.  Logistic  Concept 

e.  Operational  Mode  Sumroary/Mission  Profile 

f .  Threat 

g.  Test  Setting 

h.  Certification  for  System  Readiness  for  OT 

i.  Other  Special  Topics 

Figure  5-10.  Sample  OTRR  agenda 
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Test  Readiness  (Operational  Tester) 

a.  Test  Directorate  Organization  (Mandatory) . 

Description  of  the  overall  test  organization  and  structure  for 
the  test. 

b.  OTP  Resources/ FORSCOM  Support  (Mandatory) .  Status  of 
support  required/received  or  coordinated  in  accordance  with 
OTP. 

c.  Test  and  Evaluation  Plan/Detailed  Test  Plan 
(Mandatory) .  Overview  of  the  test  design  to  include  issues 
and  criteria  as  appropriate  and  status  of  TEP  development. 

d.  Test  Schedule  (Mandatory) 

e.  Participation/Other  Agencies 

f.  Pilot  Test  (Plan  or  Result)  (Mandatory).  Description 
of  planning  pilot  test  activities  or  results  of  the  pilot 
test. 

g.  Data  Displays 

h.  Data  Collection  Reduction  and  Processing  Plan 

i.  Test  Instrumentation  Status 

j .  ADATS 

k.  Test  Site  Support  Plan 

l.  Human  factors 

m.  Status  of  MOUs 

n.  other  Special  Topics 

5.  Overall  Readiness  (Operational  Evaluator) 

a.  Evaluator  Critique  of  System  Readiness 

b.  Evaluator  Critique  of  Tactics,  Techniques,  and  Doctrine 

c.  Evaluator  Critique  of  Threat 

d.  Evaluator  Critique  of  Training  Readiness 


Figure  5-10.  Sample  OTRR  agenda  (cont) 


e.  Evaluator  Critique  of  Test  Readiness 

f.  TEP  Status 

g.  Overall  Evaluation 

6.  DAG  Composition  and  Operation  (Operational  Tester) 

7.  ADP  Plan  (Operational  Tester) 

8.  Funding  (Operational  Tester) 

9 .  Identification  and  Review  of  Showstoppers  or  Potential 
ShowstoDPers  (Operational  Tester  and  Evaluator) 

10.  Review  of  Action  Items  (Operational  Tester) 

11.  Discussion  (All) 

12.  Decision  (Chairman)  

Figure  5-10.  Sample  OTRR  agenda  (cont) 
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TEST  EXECUTION  CHECKLIST 

1.  Prior  to  movement  to  the  field  test  site,  the  test  officer 
must  ensure  the  following  items  are  on  hand: 

a.  Test  item. 

b.  Safety  release.  Ensure  that  a  hard  copy  of  safety 
release  is  in  place  prior  to  the  start  of  any  test.  Verbal 
(telephonic)  release  in  advance  of  a  hard  copy  is 
unacceptable. 

c.  Combat  developer's  operational  test  readiness 
statement  (OTRS) . 

d.  Materiel  developer's  OTRS. 

e.  Training  developer's  OTRS. 

f.  Complete  system  support  package. 

g.  Sufficient  copies  of  data  collection  and  data 
reduction  forms. 

2.  Test  site  activities: 

a.  Ensure  test  support  contractor  sets  up  support  site  in 
accordance  with  test  support  plan. 

b.  Establish  a  field  test  operation  center  (FTOC) . 

(1)  Establish  access  roster  of  essential  personnel 
who  can  enter  FTOC. 


(2)  Maintain  a  daily  log  of  activities  in  FTOC  for 
the  pilot  test  and  the  test. 

(3)  Provide  parent  test  directorate  with  a  daily 
activities  report  at  the  end  of  each  test  day. 

(4)  Always  have  someone  knowledgeable  at  FTOC  during 
testing  activities. 

(5)  Ensure  FTOC  area  is  policed,  and  when  clearing 
area  leave  it  better  than  you  found  it. 


Figure  5-11.  Test  execution  checklist 


c.  Establish  a  facility  (porta-kamp  or  tent)  to  conduct 
briefings  and  pretest  training  and  to  administer 
questionnaires . 

d.  Establish  a  facility  with  telephone  for  the  developers 
and  proponent  (limit  their  access  to  field  test  location 
without  escort) . 

e.  Establish  a  data  collection-data  reduction  facility. 

(1)  Establish  access  roster  of  essential  personnel 
who  can  enter  data  collection-data  reduction  facility. 

(2)  Maintain  continuous  quality  control  of  data  and 
if  data  collection  is  not  done  right  ensure  that  event  is  done 
again. 

f.  Ensure  that  video  documentary  script  outline  is 
adhered  to  in  getting  video  and  photographic  coverage  of  test. 

3.  Pretest  activities. 

a.  Final  coordination  with  test  support  unit  will  include 
the  following  actions: 

(1)  Organize  test  support. 

(2)  Coordinate  movement  of  unit  to  field  test  site. 

(3)  Establish  rapport  with  test  unit  commander  and 
his  parent  organization. 

b.  Prepare  test  equipment  for  testing. 

c.  Prepare  visitor's  briefing. 

d.  Set  up  instrumentation  and/or  special  equipment  and 
ensure  it  is  operational. 

e.  Conduct  training  for  players  (both  individual  and 
unit),  data  collectors-reducers,  and  controllers-evaluators. 

f.  Conduct  vehicle  safety  class  and  establish  test 
support  vehicles  control  procedures. 

g.  Conduct  overall  safety  class. 

4.  Pilot  test  to  exercise  the  following  requirements  of  the 
test: 


Figure  5-11  (cont) .  Test  execution  checklist 
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a.  Instrumentation. 


b.  Control  procedures. 

c.  Data  collection. 

d.  Data  reduction. 

e.  Players. 

f .  Test  support . 

g.  Scenario. 

5.  Conduct  pilot  test  in-process  review  and  corrective 
action. 

6.  Test. 

a.  Collect  data. 

b.  Start  data  reduction. 

c.  Start  data  input  into  computer. 

d.  Report  all  incidents  in  compliance  with  AR  73-XX  on 
the  forms  reflected  in  DA  Pam  73-XX.  See  DA  Pam  73-XX  for 
distribution  guidance. 

e.  Supervise  property  accountability,  ensuring  that 
statements  are  made  when  the  event-incident  occurs. 

f.  Seek  assistance  in  keeping  add-on  testing  to  a  minimum 
or  forbidding  it. 

g.  Emphasize  the  sequence  of  events  frequently  to  the 
test  unit  commander  and  his  S3. 

h.  Document  deviations  from  detailed  test  plan. 

i.  Document  problems  and  submit  an  after  action  report. 

j.  Conduct  a  midtest  review. 

k.  Conduct  visitor  briefings  as  appropriate. 

Figure  5-11  (cont) .  Test  execution  checklist 
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7 .  End  test . 

a.  Clean  up  and  return  test  items. 

b.  Clean  up  and  turn  in  equipment. 

c.  Clear  up  all  hand  receipts. 

d.  Release  support  personnel  after  all  equipment  has  been 
turned  in. 

e.  Release  data  collectors-evaluators  only  after  ensuring 
that  all  data  is  in  and  quality  checked  by  the  data  manager. 

f.  Prepare  recognition  for  test  participants. 

g.  Finalize  test  reports. 


Figure  5-11  (cont) .  Test  execution  checklist 


LEVEL 

DESCRIPTION 

POSSIBLE  FORMS 

EXAHPLES  OF  CONTENT 

DISPOSITION 

Level  1 

Data  in  their 

Complete  data 

1.  All  reported  target 

Accumulated 

data  “raw 

original  form. 

collection  sheets, 

presentations  and 

during  trials 

data“ 

Results  of  field 
trials  just  as 
recorded. 

exposed  camera  film, 
voice  recording 
tapes,  original 
instrunentation 
magnetic  tape  or 
printouts,  original 
videotapes,  filled 
questionnaires, 
interview  notes. 

detection. 

2.  Clock  times  of  all 
events. 

3.  Azimuth  and  vertical 
angle  from  each 
flash  base  for  each 
flash. 

4.  Recording  tapes  of 
interviews. 

for  processing. 

Usual ly 

discarded  after 
use.  Not 
ordinari  ly 
given  to 
another  agency. 

Mot  published. 

Level  2 

Data  taken  from 

Confirmed  and 

1.  Record  of  all  valid 

Produced  during 

data 

the  raw  form  and 

corrected  data 

detections. 

processing. 

“reduced 

consolidated. 

collection  sheets. 

2.  Start  and  stop  times 

Usual ly 

data“ 

Invalid  or 
unnecessary  data 
points  deleted. 
Trials  declared 
“no  test" 
deleted. 

film  with  extraneous 
footage  deleted, 
corrected  tapes  of 
printouts,  and 
original  raw  data 
with  “no  test"  events 
marked  out. 

of  all  applicable 
events. 

3.  Computed  impact 
points  of  each  round 
flashed. 

4.  Confirmed  interview 
records . 

discarded  after 
use.  Not 
published. 

Level  3 

Data  which  have 

Spread  sheets. 

1.  Counts  of  detections 

Not  usually 

data 

been  checked  for 

tables,  typed  lists, 

arranged  in  sets 

published  but 

“ordered 

accuracy  and 

ordered  and  labeled 

showing  conditions 

made  available 

data“ 

arranged  in 
convenient  order 
for  handling. 
Operations 
limited  to 
counting  and 
elementary 
arithmetic. 

printouts,  purified 
and  ordered  tape, 
edited  film,  edited 
magnetic  tapes, 
ordered  punch  cards. 

under  which 
detections  occurred. 

2.  Elapsed  times  by 
type  events. 

3.  Im^ct  points  of 
rounds  by  condition 
under  which  fired. 

4.  Interview  comments 
categorized  by  type. 

to  analysts. 

Usually  stored 
in 

institutional 
data  banks. 

All  or  part  may 
be  published  as 
supplements  to 
test  report. 

Level  4 

Data  which  have 

Tables  or  graphs 

1.  Percentage  of 

Published  as 

data 

been  simmarized 

showing  totals, 

presentations 

the  basic 

“findings” 

by  elementary 

means,  medians, 

detected. 

factual 

or 

mathematical 

modes,  maxinuns, 
minimums,  quartiles, 
deci les, -percentiles, 
curves,  or  standard 
deviations. 

Qualitative  data  in 
form  of  lists, 
histographs,  counts 
by  type,  or  surmary 
statements. 

2.  Mean  elapsed  times. 

findings  of 

“sunmary 

statis¬ 

tics*' 

operations. 
Operations 
limited  to 
descriptive  sum¬ 
maries;  no 
Judgments  or 
inferences.  Does 
not  go  beyond 
what  was  observed 
in  test. 

3.  Calculated  probable 
errors  about  the 
centers  of  impact  or 
conditions. 

4.  Bar  graph  showing 
relative  frequency 
of  each  category  of 
comment. 

test  report. 

Figure  5-12.  Levels  of  data 
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Level  5 
data 

••analysis*' 
or  “infer¬ 
ential 
statis¬ 
tics” 


Level  6 
data 

“extended 

analysis” 

or 

operations 


DESCRIPTION 

Data  resulting 
from  statistical 
tests  of 
hypothesis  or 
interval 
estimation. 
Execution  of 
planned  analysis 
data.  Includes 
both  comparisons 
and  statistical 
significance 
level.  Judgments 
limited  to 
analysts 
selection  of 
technics  and 
significant 
levels. 


Data  resulting 
from  further 
analytic 
treatment  going 
beyond  primary 
statistical 
analysis, 
combination  of 
analytic  results 
from  different 
sources,  or 
exercise  of 
simulation  or 
models. 

Judgments  limited 
to  analysts' 
choices  only. 


POSSIBLE  FORMS 

Results  of  primary 
statistical 
techniques  such  as 
T- tests.  Chi-square, 
F-test,  analysis  of 
variance,  regression 
analysis,  contingency 
table  analyses  and 
other  associated 
confidence  levels. 
Follow-on  tests  of 
hypotheses  arising 
from  results  of 
earlier  analysis,  or 
fallback  to  alternate 
nonparametric 
technique  when 
distribution  of  data 
does  not  support 
assumption  of 
normality. 

Qualitative  data  in 
the  form  of 
prevailing  consensus. 

Insertion  of  test 
data  into  a 
computational  model 
or  a  combat  simu¬ 
lation,  aggregation 
of  data  from 
different  sources 
observing  required 
disciplines,  curve 
fitting  and  other 
analytic 

generalization,  or 
other  operations 
research  techniques 
such  as  application 
of  queuing  theory, 
inventory  theory, 
cost  analysis,  or 
decision  analysis 
techniques. 


EXAMPLES  OF  CONTENT 

1.  Inferred  probability 
of  detection  with 
its  confidence 
interval. 

2.  Significance  of 
difference  between 
two  mean  elapsed 
times. 

3.  Significance  of 
difference  between 
observed  protoblc 
error  and  criterion 
threshold. 

4.  Magnitude  of 
difference  between 
categories  of 
comments. 


Level  7 

Data  conclusions 

Stated  conclusions  as 

1. 

data 

resulting  from 

to  issues,  position 

“conclu¬ 

applying 

statements. 

sions”  or 

evaluative 

challenges  to 

evaluation 

military 
judgments  to 
analytic  results. 

validity  or  analysis. 

2. 

3. 

DISPOSITION 

published  in 

evaluation 

reports. 

(If  evaluation 
report  is  part 
of  test  report, 
the  level  5 
analysis 
results  are 
presented 
separately  from 
the  level  4 
findings.) 


1.  Computation  of 
probability  of  hit 
based  on  target 
detection  data  from 
test  combined  with 
separate  data  or 
probability  of  hit 
given  a  detection. 

2.  Exercise  of 
attrition  model 
using  empirical  test 
times  distribution. 

3.  Determination  of 
whether  a  trend  can 
be  identified  from 
correlation  of  flash 
base  accuracy  data 
under  stated 
conditions  from 
different  sources. 

4.  Delphi  technique 
treatment  of 
consensus  of 
interview  comments. 


Conclusion  as  to 
whether  probability 
of  detection  is 
adequate. 

Conclusion  as  to 
timeliness  of  system 
performance. 
Conclusion  as  to 
military  value  of 
flash  base  accuracy. 
Conclusion  as  to 
main  problems 
identified  by 
interviewees. 


Published  as 
appropriate  in 
evaluation 
reports. 


Published  as 
the  basic 
evaluative 
conclusions  of 
evaluation 
reports. 


Figure  5 —I 2  ( cont ) .  Levels  of  data 
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Chapter  6 

Operational  Test  and  Evaluation  Reporting 


Section  I 
General 


This  chapter  provides  policy  and  procedural  guidance  for  the  test 
officer  and  the  evaluator  on  test  reporting,  system  evaluations, 
and  system  assessments.  This  chapter  also  provides  guidance  and 
suggestions  for  the  test  officer  and  the  evaluator  when  preparing 
a  report  of  test,  evaluation,  or  assessment. 

6-2.  Scope  .  ,  j 

This  chapter  addresses  the  following  reporting,  evaluation,  and 
assessment  documents;  Test  and  Evaluation  Reports  (TER), 
Operational  Assessments  (OA) ,  Early  Operational  Assessments 
(EOA) ,  Abbreviated  Operational  Assessments  (AOA) ,  Test  Reports 
(TR) ,  Preliminary  Test  and  Evaluation  Reports-Test  Design  and 
Data  Base  (PTER-TDDB) ,  and  Preliminary  Operational  Assessments- 
Test  Design  and  Data  Base  (POA-TDDB) . 

6-3.  Applicability 

This  chapter  applies  to  OTE  of  materiel  systems  and  Information 
Mission  Area  (IMA)  systems,  and  to  user  tests  of  combat  and 
training  developments  products. 


Section  II 
Mission 


6-4.  Test  mission 

Conducting  tests  is  the  primary  mission  of  the  Operational  Tester 
(OT) .  The  term  OT  is  used  throughout  this  chapter  to  describe 
the  test  organization,  normally  TEXCOM.  The  term  "tester” 
describes  the  test  officer  or  test  director  responsible  to  the  OT 
for  test  conduct.  The  term  "test  analyst"  describes  the 
individual (s)  that  provide  analytic  support  to  the  tester. 

6-5.  Evaluation  mission 

Evaluating  or  assessing  the  system  operational  effectiveness  and 
suitability  using  the  results  of  testing  is  the  primary  mission 
of  the  lOE.  The  term  lOE  is  used  throughout  this  chapter  to 
describe  the  evaluation  organization,  normally  OEC.  The  term 
"evaluator"  describes  the  individual  responsible  to  the  lOE  for 
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system  evaluation.  The  term  "analyst”  describes  the 
individual (s)  that  provide  analytic  support  to  the  evaluator. 


Section  III 
Reports 


6-6.  General  .  ... 

The  end  product  of  the  test  mission  and  the  evaluation  mission  is 
a  report  and  subsequent  briefinq  and  recommendations  to  the 
Materiel  Acquisition  Process  (MADP) /Major  Automated  Information 
System  Review  Council  (MAISRC)  decision  body.  The  approved 
report  is  distributed  throughout  the  Army  and  other  Department  of 
Defense  (DOD)  elements. 

6-7.  Reporting  standards 

The  report  must  be  timely,  well  written,  and  contain  all  the 
information  necessary  to  make  high-level  decisions.  A  defective 
report  loses  the  validity  and  reliability  of  a  test  even  though 
the  test  may  be  well-conducted  and  the  evaluation  well  reasoned 
and  thorough. 

6-8 .  Report  preparation 

Throughout  the  OTE  cycle,  utilize  all  previous  efforts  on 
planning,  testing,  and  reporting  in  the  preparation  of  a  TER  or 
other  reporting  document.  The  report  requires  certain 
introductory  and  explanatory  material  to  make  it  a  stand-alone 
document  and  varies  substantially  in  size  and  form  depending  on 
the  size  and  scope  of  the  test,  evaluation,  or  assessment. 


Section  IV 
Reporting  Documents 


6-9.  Use  of  evaluations  and  assessments 

Evaluations  and  assessments  provide  formal  analyses  of  the 
system's  operational  effectiveness  and  suitability  to  decision 
makers  at  milestone  decision  review  (MDR) .  Assessments  may  also 
provide  periodic  status  reports  throughout  the  life  cycle  of  a 
gystem  whenever  continuous  evaluation  identifies  changes 
occurring  in  the  status  of  the  system. 

6-10.  Purpose  of  TER,  OA,  EOA,  and  AOA 

These  document  each  and  every  independent  operational  evaluation 
or  assessment.  They  address  and  answer  the  critical  and 
additional  issues  in  the  TEP  based  on  all  available  data  and 
analytic  treatment  of  that  data. 
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6-11.  Differences  between  evaluations  and  assessments 
In  evaluations,  the  lOE  draws  conclusions  on  system  operational 
effectiveness  and  system  operational  suitability.  In 
assessments,  the  lOE  makes  findings  on  progress  towards  required 
effectiveness  and  suitability  and  the  adequacy  of  system 
programmatics  in  place  to  meet  the  requirements  at  a  given  future 
point. 

6-12.  Content  of  TER,  OA,  EOA,  and  AOA 

Evaluations  and  assessments  contain  analyses  and  supporting 
rationale  for  the  conclusions,  data  summaries,  positions  on  the 
future  capability  of  the  system  to  fulfill  approved  requirements, 
program  constraints,  and  impacts  on  the  evaluation.  The  final 
draft  report  is  provided  to  the  MDR  prior  to  the  decision  meeting 
in  accordance  with  AR  70-1  or  AR  25-3.  AOA  may  be  used  to  update 
positions  as  required. 

6-13.  Test  reporting 

In  addition  to  its  evaluation  function,  the  TER,  OA,  or  EOA  is 
also  the  primary  record  of  the  operational  test,  i.e.,  EUTE,  LUT, 
lOT,  and  FOT.  Every  operational  test  concludes  with  a  TER,  OA, 
or  EOA.  Other  user  tests  (FDTE,  CEP,  CT)  conclude  with  a  TR. 
Operational  testing  supports  operational  evaluations  or 
assessments  by  generating  relevant  data  not  obtainable  through 
other  sources. 

6-14.  Test  report  content 

a.  The  report  of  test,  incorporated  into  the  TER,  OA,  or  EOA, 
presents  the  factors  and  conditions  used  in  conduct  of  the  test, 
documents  the  data  base  created  in  the  test,  and  presents  the 
level  3  and  higher  data  (both  qualitative  and  quantitative) 
resulting  from  the  test.  The  report  requires  certain 
introductory  and  explanatory  material  and  varies  substantially  in 
size  and  form  depending  on  the  test.  In  every  case,  it  fully  and 
accurately  states  what  was  done  and  what  resulted. 

b.  Any  departures  from  the  TEP,  as  well  as  unexpected 
conditions  encountered,  are  reported  along  with  explanations 
necessary  to  evaluate  the  adequacy  and  validity  of  the  test. 

Test  data  are  limited  to  findings  of  fact  rather  than 
conclusions,  and  therefore,  reporting  of  data  and  test 
occurrences  are  not  subject  to  concurrence  by  any  other  activity. 
No  approving  echelon  has  the  authority  to  change  the  reported 
facts. 

6-15.  AE  system  tests 

The  TER  for  AE  systems  documents  tester answers  to  the  critical 
and  additional  issues  in  the  TEP  based  on  data  from  the  test  and 
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analytic  treatment  of  that  data.  In  these  reports,  the  OT  draws 
conclusions  on  system  operational  effectiveness  and  suitability 
based  solely  on  the  results  of  the  user  test. 

6-16.  Other  user  tests  (FDTE,  CEP,  CT) 

The  TR  for  FDTE,  CEP,  and  CT  documents  answers  to  the  user  issues 
in  the  TEP  based  on  available  data  and  develops  and  documents 
conclusions  derived  from  the  answers  to  the  issues  by  the  tester. 

6-17.  Preliminary  reports 

For  full-evaluate  systems,  the  OT  prepares  and  approves  a 
preliminary  report  (PTER-TDDB  or  POA-TDDB) .  This  report  provides 
a  description  of  the  test  and  the  level  3  data  base  to  ^ interested 
DA  and  DOD  elements.  The  preliminary  report  is  a  partial  report 
which  is  produced  as  a  part  of  the  development  of  the  complete 
TER,  OA,  or  EOA.  There  is  no  additional  effort  required  to  ^ 
produce  this  report.  This  report  is  not  an  interim  report,  it 
(jraws  no  conclusions,  and  it  provides  no  analysis.  It  is  merely 
a  vehicle  to  get  final  test  data  to  the  program  manager, 
technical  evaluator,  DA  staff,  and  DOTE. 


Section  V 

Executive  Summaries 


6-18.  Executive  summary  purpose 

Executive  summaries  of  TER,  OA,  and  EOA  provide  the  essence  of 
the  primary  document  to  other  members  of  the  acquisition  team  and 
decision  makers.  Executive  summaries  also  provide  advance  notice 
of  the  content  of  the  document  while  it  goes  through  the  review, 
approval,  and  publication  process. 

6-19.  Requirements 

The  executive  summary  is  only  required  if  the  body  of  the  TER/TR 
exceeds  twenty-five  pages.  If  the  volume  of  the  TER  is  less  than 
twenty- five  pages,  an  executive  summary  is  optional.  The 
executive  summary  is  also  optional  for  all  CEP  and  CT. 

6-20.  Executive  summary  writing 

a.  The  evaluator  and  tester  jointly  prepare  the  executive 
summary  for  all  TER  for  full-evaluate  systems  and  for  all  OA/EOA 
for  full-evaluate  systems  (with  user  tests) .  The  evaluator  has 
the  lead  for  the  executive  summary  and  assigns  the  tester  (as  a 
member  of  the  system  task  force)  to  write  pertinent  portions  of 
the  executive  summary  as  appropriate.  The  evaluator  prepares  the 
executive  summary  in  its  entirety  for  all  OA/EOA  for  full— 
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evaluate  systems  (with  no  user  test) .  The  tester  prepares  the 
executive  summary  for  all  TER  (AE)  and  TR  for  FDTE,  CEP,  and  CT. 

b.  The  executive  summary  provides  an  overview  of  the  test  and 
of  the  evaluation/assessment  and  their  results,  and  should  be 
written  after  the  draft  of  all  the  chapters  of  the  report  are 
written.  The  executive  summary  is  to  be  written  for  senior 
officials  and  decision  makers  and  is  to  be  on  the  order  of  30 
pages  or  less.  In  general,  smaller  is  better.  It  may  be 
detached  from  the  remainder  of  the  report  and  is  therefore  to  be 
written  so  that  it  can  stand  alone. 

c.  When  forwarding  the  executive  summary  to  the  decision 
makers  as  a  separate  document,  append  the  signature  block  and  the 
signature  of  the  Commander,  OPTEC  to  the  end  of  the  executive 
summary . 

6-21.  Executive  summary  standards. 

a.  A  good  executive  summary  avoids  repeating  verbatim  entire 
sections  from  the  remainder  of  the  report.  It  is  a  comprehensive 
and  brief  abstract,  recapitulation  or  compendium  of  facts  or 
statements  from  the  body  of  the  report  referring  the  reader  to 
the  report  body  or  appendixes,  where  appropriate,  for  more 
details  or  information. 

b.  The  executive  summary  shall  contain  only  information  which 
is  reported  in  the  body  of  the  report  and  reference  the  charts, 
graphs,  illustrations,  and  photographs  as  much  as  possible  to 
present  the  essential  information  briefly  and  concisely. 


Section  VI 

Other  Reporting  Media 


6-22.  Independent  evaluator  briefings  (lEB) 

a.  lEB  based  on  the  TER,  OA,  or  EOA,  the  advance  executive 
summary  of  these  documents,  or  emerging  results  provide  input  as 
necessary  to  members  of  the  acquisition  team  and  to  decision 
makers.  lEB  follow  the  same  outline  as  the  executive  summary. 

b.  Evaluators  prepare  formal  lEB  and  present  these  briefings 
to  the  decision  bodies  at  each  MDR.  The  briefing  summarizes  the 
report  submitted  to  the  body  (DAB,  ASARC,  MAI SRC,  or  IPR  panel) 
and  contributes  to  recommendations  by  the  review  body  to  the 
decision  maker  as  well  as  to  management  decisions  by  the  review 
body. 
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6-23.  Cover  memoranda 

a.  The  lOE  forwards  reports  and  assessments  to  the  decision 
body  (ASARC,  MAISRC,  IPR,  or  MRRB)  by  a  memorandum  prepared  by 
the  evaluator.  The  memorandum  contains  transmittal  information 
and  the  final  OPTEC  ASARC/MAISRC/IPR/MRRB  position.  Full 
evaluate  system  memoranda  will  be  signed  by  CG,  OPTEC.  AE 
memoranda  will  be  signed  by  the  lOE  commander. 

b.  The  OT  forwards  TR  to  TRADOC  using  a  cover  memorandum 
prepared  by  the  tester  and  signed  by  the  OT  commander.  The 
tester  prepares  memoranda  for  CG,  TEXCOM  to  forward  preliminary 
reports  (PTER-TDDB,  POA-TDDB)  for  full  evaluate  to  acquisition 
team  members  as  specified  by  OEC. 


Section  VII 

Full  Evaluation  Reports 


6-24.  General 

Only  the  documents  described  in  this  chapter  will  be  used  to 
report  the  results  of  operational  testing,  operational 
evaluations,  and  operational  assessments.  Figure  6-1  is  a 
graphic  decision  chart  to  be  used  to  select  the  appropriate 
document . 

(INSERT  FIGURE  6-1) 


6-25.  Format 

Reporting  documents  use  a  single,  standard  format  that  may  be 
easily  tailored  to  all  of  the  various  documents  required  by  this 
chapter.  Figure  6-2  shows  this  format. 

(INSERT  FIGURE  6-2) 

6-26.  The  TER  for  full  evaluate  systems 

a.  The  evaluator  and  the  tester  jointly  write  a  TER  (using 
the  format  in  Figure  6-2)  for  all  full-evaluate  systems  after 
materiel /IMA  life  cycle  user  tests  of  the  complete  system  to 
include  lOT  and  those  FOT  where  the  test  is  conducted  in  the  same 
manner  as  an  lOT  (full  FOT) .  The  title  of  the  TER  implies  to  the 
decision  maker  or  other  reader  that  the  evaluation  is  based  on  a 
full  operational  test  of  the  entire  system  (as  well  as  the 
results  of  FDTE,  CEP,  CT,  DT,  contractor  tests,  market  surveys 
and  investigations,  modeling  and  simulation,  or  studies  and 
analyses) .  The  TER  is  the  primary  record  of  the  operational  test 
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and  the  operational  evaluation  supporting  the  MADP  or  MAI SRC 
decision. 

b.  The  tester  prepares  chapters  1  and  2  of  the  TER  with  input 
and  assistance  from  the  evaluator  who  is  part  of  the  test  team. 
The  tester  prepares  the  PTER-TDDB  based  on  chapters  1  and  2 . 
Figure  6-3  provides  the  format  for  a  PTER-TDDB.  The  evaluator 
prepares  chapters  3  and  4  with  input  and  assistance  from  the 
tester  who  is  part  of  the  system  task  force.  For  some 
appendixes,  the  evaluator  has  the  lead.  For  the  remainder  of  the 
appendixes,  the  tester  has  the  lead. 

(INSERT  FIGURE  6-3) 

c.  A  TER  reports  the  results  of  a  dedicated  phase  Of 
operational  testing  and  the  evaluation  of  the  system  operational 
effectiveness  and  suitability.  Evaluations  are  based  on  results 
of  the  operational  test  plus  available  technical,  contractor,  or 
other  user  (FDTE,  CEP,  CT)  testing.  Evaluations  consider  results 
of  modeling,  simulation,  market  surveys  or  investigations,  and 
any  other  studies  or  analyses. 

d.  The  report  documents  the  test  and  the  evaluation  and 
contains  an  analysis  addressing  the  issues  involved,  the  lOE 
conclusions,  major  findings,  analyses  and  supporting  rationale 
for  the  conclusions,  a  position  on  the  future  capability  of  the 
system  to  fulfill  the  approved  requirements,  program  constraints 
and  their  impact  on  the  evaluation,  and  lOE  recommendations. 

e.  CG,  OPTEC  approves  and  releases  the  TER  and  signs  the 
cover  memorandum  sending  the  TER  to  the  ASARC,  MAISRC,  or  IPR. 
Format  for  cover  memoranda  is  at  figure  6-4.  The  lOE  and  OT 
commanders  also  sign  the  TER.  The  tester  commander  approves  the 
release  of  PTER-TDDB  and  signs  a  transmittal  memorandum. 

(INSERT  FIGURE  6-4) 

6-27.  OA  or  EOA  for  full  evaluate  systems  (after  a  dedicated 
phase  of  operational  testing) 

a.  An  OA  or  EOA  following  a  user  test  uses  the  same  format 
and  procedures  as  a  TER.  (See  figure  6-2.)  Assessments  differ 
from  the  TER  only  in  the  title  of  the  report  and  the  designations 
of  some  paragraphs  within  the  report.  The  only  difference 
between  an  OA  and  an  EOA  is  that  prior  to  MS  II,  designate  the 
document  as  an  EOA.  After  MS  II,  designate  the  document  as  an 
OA. 
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b.  The  evaluator  and  the  tester  jointly  write  an  OA  for  all 
full-evaluate  systems  after  materiel/IMA  life  cycle  user  tests  of 
less  than  the  complete  system  to  include  LUT  and  those  FOT  where 
the  test  has  been  tailored  to  something  less  than  an  lOT 
equivalent  (i.e.,  it  has  been  tailored  to  resemble  an  LUT).  The 
evaluator  and  the  tester  jointly  write  an  EOA  for  all  full- 
evaluate  systems  after  early  materiel/IMA  life  cycle  user  tests 
of  less  than  the  complete  system  to  include  BUT,  EUE,  and  those 
FDTE  or  CEP  conducted  in  lieu  of  EUTE. 

c.  The  title  of  the  OA  or  EOA  implies  to  the  decision  maker 
or  other  reader  that  the  assessment  is  based  on  a  less  than  full 
operational  test  of  the  entire  system  and  lOE  "conclusions'  are 
assessments  of  progress  toward  system  requirements  and  goals  and 
findings  on  the  adequacy  of  programmatics  toward  this  end.  The 
results  of  FDTE,  CEP,  CT,  DT,  contractor  testing,  market  surveys 
and  investigations,  modeling  and  simulation,  or  studies  and 
analyses  are  also  included  in  the  assessment. 

d.  The  OA  or  EOA  is  the  primary  record  of  an  operational  test 
and  the  operational  assessment  supporting  a  MADP  or  l^ISRC 
decision.  Every  EUTE,  LUT,  and  every  FOT  conducted  in  the  same 
manner  as  a  LUT  (abbreviated  FOT)  concludes  with  a  OA  or  EOA. 

e.  The  tester  prepares  chapters  1  and  2  of  the  OA/EOA  with 
input  and  assistance  from  the  evaluator  who  is  part  of  the  test 
team.  The  tester  prepares  the  POA-TDDB  based  on  chapters  1  and  2 
(using  the  format  in  figure  6-3) .  The  evaluator  prepares 
chapters  3  and  4  with  input  and  assistance  from  the  tester  who  is 
part  of  the  system  task  force.  For  some  appendixes,  the 
evaluator  has  the  lead.  For  the  remainder  of  the  appendixes,  the 
tester  has  the  lead. 

f.  OA  or  EOA  report  the  results  of  a  dedicated  phase  of 
operational  testing  and  assessment  of  the  system  operational 
effectiveness  and  suitability.  Assessments  are  based  on  results 
of  the  operational  test  plus  available  development,  contractor, 
or  other  user  (FDTE,  CEP,  CT)  testing.  Assessments  consider 
results  of  modeling,  simulation,  market  surveys  or 
investigations,  and  any  other  studies  or  analyses. 

g.  The  report  documents  the  assessment  and  contains  an 
analysis  addressing  the  issues  involved,  the  lOE  conclusions, 
major  findings,  analyses  and  supporting  rationale  for  the 
conclusions,  a  position  on  the  future  capability  of  the  system  to 
fulfill  the  approved  requirements,  program  constraints  and  their 
impact  on  the  assessment,  and  lOE  recommendations. 
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h.  CG,  OPTEC  approves  and  releases  the  OA/EOA  and  signs  the 
cover  memorandum  sending  the  OA/EOA  to  the  ASARC,  MAISRC,  or  IPR. 
Format  for  cover  memoranda  is  at  figure  6-4.  The  lOE  and  OT 
commanders  also  sign  the  OA/EOA.  The  tester  commander  approves 
the  release  of  POA-TDDB  and  signs  the  transmittal  memorandum. 

6-28.  OA  or  EOA  for  full  evaluate  systems  (with  no  dedicated 
phase  of  operational  testing) 

a.  An  OA  or  EOA  with  no  user  test  uses  a  tailored  TER  format 
and  procedures.  (See  figure  6-2.)  These  assessments  differ  in 
the  designations  and  omissions  of  some  paragraphs  and  appendixes 
within  the  report.  The  only  difference  between  an  OA  and  an  EOA 
is  that  prior  to  MS  II,  designate  the  document  as  an  EOA.  After 
MS  II,  designate  the  document  as  an  OA. 

b.  The  OA  or  EOA  is  the  primary  record  of  an  operational 
assessment  supporting  a  MADP/MAISRC  decision  where  no  EUTE,  LUT, 
lOT,  or  FOT  has  been  conducted.  The  evaluator  prepares  all  parts 
of  the  OA/EOA. 

c.  The  evaluator  writes  an  OA  or  an  EOA  for  all  full-evaluate 
systems  prior  to  all  materiel/IMA  life  cycle  decision  points  not 
supported  by  EUTE,  LUT,  lOT,  or  FOT  (i.e.,  supported  by  DT, 
contractor  testing,  market  surveys  and  investigations,  modeling 
and  simulation,  studies  and  analyses,  or  combat/ training 
developer  planned  FDTE,  CEP,  or  CT) .  The  evaluator  also  writes 
an  OA  or  EOA  to  report  the  results  of  CE  to  the  decision  maker 
when  CE  events  dictate  the  need  for  a  report  prior  to  the  next 
milestone. 

d.  The  title  of  the  OA  or  EOA  implies  to  the  decision  maker 
or  other  reader  that  the  assessment  is  based  on  a  less  than  full 
operational  test  of  the  entire  system  and  lOE  ••conclusions"  are 
assessments  of  progress  toward  system  reguirements  and  goals  and 
findings  on  the  adequacy  of  programmatics  toward  this  end. 

e.  An  OA  or  EOA  reports  the  assessment  of  the  system 
operational  effectiveness  and  suitability.  Assessments  are  based 
on  results  of  available  development,  contractor,  or  other  user 
(FDTE,  CEP,  CT)  testing.  Assessments  consider  results  of 
modeling,  simulation,  market  surveys  or  investigations,  and  any 
other  studies  or  analyses. 

f.  The  report  documents  the  assessment  and  contains  an 
analysis  addressing  the  issues  involved,  the  lOE  conclusions, 
major  findings,  analyses  and  supporting  rationale  for  the 
conclusions,  a  position  on  the  future  capability  of  the  system  to 
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fulfill  the  approved  requirements,  program  constraints  and  their 
impact  on  the  assessment,  and  lOE  recommendations. 

g.  CG,  OPTEC  approves  and  releases  the  OA/EOA  and  signs  the 
cover  memorandum  sending  the  OA/EOA  to  the  ASARC,  MAISRC,  or  IPR. 
Format  for  cover  memoranda  is  at  figure  6-4.  The  lOE  commander 
also  signs  the  OA/EOA.  There  is  no  preliminary  report. 


Section  VIII 

Abbreviated  Evaluation  Reports 


6-29.  TER(AE) 

a.  For  AE  systems,  the  operational  test  provides  all  the  data 
and  insight  required  to  draw  analytic  conclusions  on 
effectiveness  and  suitability.  The  tester  writes  a  TER{AE)  for 
all  AE  systems  after  any  and  all  materiel/IMA  life  cycle 
operational  tests  to  include  EUTE,  LUT,  lOT,  and  FOT.  A  TER(AE) 
contains  the  report  of  test  plus  analytic  findings  and 
conclusions  on  operational  effectiveness  and  suitability  by  the 
tester.  The  format  of  a  TER(AE)  is  shown  in  figure  6-2. 

b.  This  TER(AE)  is  the  primary  record  of  an  operational  test 
for  an  AE  system.  Every  operational  test  for  an  AE  system 
concludes  with  a  TER(AE) . 

c.  For  AE  systems,  the  tester  prepares  the  TER  in  its 
entirety.  The  OT  commander  signs  the  report  and  releases  it  to 
the  OEC  commander  for  transmittal  to  the  IPR.  There  is  no  PTER- 
TDDB  or  POA-TDDB  for  AE  systems. 

6-30.  AOA  for  full  evaluate  systems 

a.  The  evaluator  writes  an  AOA  for  all  full-evaluate  systems 
prior  to  any  and  all  materiel/IMA  life  cycle  decision  points  not 
otherwise  supported  by  a  TER,  OA,  or  EOA.  The  format  for  an  AOA 
is  shown  in  figure  6-5. 

(INSERT  FIGURE  6-5) 

b.  The  primary  use  of  the  AOA  for  these  systems  is  at  MS  I 
and  at  Materiel  Release  Decisions  subsequent  to  MS  III.  The  AOA 
may  also  be  used  to  report  a  CE  event  between  milestones  in  lieu 
of  a  full  OA  or  EOA.  The  Commander,  OEC  approves  the  AOA  and 
signs  the  cover  memorandum  sending  the  AOA  to  Commander,  OPTEC. 
Format  for  cover  memoranda  is  at  figure  6-4. 
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c.  The  AOA  is  used  when  little  or  no  new  information  on  the 
system  is  available  and  contains  summary  assessments  of 
operational  effectiveness  and  suitability  by  the  evaluator. 

6-31.  AOA  for  AE  systems 

a.  The  evaluator  writes  an  AOA  for  all  AE  systems  prior  to 
any  and  all  materiel/IMA  life  cycle  decision  points  using  the 
format  in  figure  6-3.  Where  a  TER(AE)  is  available  to  support 
the  decision  point,  the  AOA  contains  confirmation  of  tester 
evaluative  findings  by  the  evaluator.  The  TER(AE)  will  be 
provided  to  the  decision  maker  along  with  this  AOA. 

b.  Where  a  TER(AE)  is  not  available  to  support  the  decision 
point,  the  AOA  contains  summary  assessments  of  operational 
effectiveness  and  suitability  by  the  evaluator.  A  combined  TT/OT 
report  may  be  referenced  in  the  AOA. 

c.  The  OEC  commander  approves  the  AOA  and  signs  the  cover 
memorandum  sending  the  AOA  (and  any  available  TER(AE))  to  the 
Commander,  OPTEC.  Format  for  cover  memoranda  is  at  Figure  6-4. 
There  is  no  preliminary  report. 


Section  IX 
Other  Reports 


6-32.  TR 

a.  The  TR  is  the  primary  record  of  the  user  test  of  a 
materiel  or  IMA  system  or  a  combat/training  developments  product 
(i.e.,  doctrine,  training,  organization,  leader  development, 
materiel  requirements)  conducted  as  a  FDTE,  CEP,  or  CT.  Each  of 
these  user  tests  concludes  with  a  TR.  The  tester  prepares  the  TR 
in  its  entirety  using  the  format  in  figure  6-2. 

b.  The  TR  is  written  by  the  tester  after  FDTE,  CEP,  and  CT. 
This  TR  contains  the  report  of  test  by  the  tester  for  the  test 
sponsor . 

c.  The  report  may  or  may  not  include  evaluative  findings  of 
the  tester  depending  on  the  requirements  of  the  test  sponsor. 
lOE  involvement  is  only  applicable  where  the  results  may  be  used 
in  CE  or  in  evaluation  or  assessment  reports. 

6-33.  Combined  DT  and  OT 
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a.  When  the  DT  and  the  OT  are  combined  into  a  single  test 
which  substitutes  for  lOT  or  full  FOT,  a  TER  is  written  on  the 
operational  portions  of  the  test.  lOE  evaluation  also  includes 
the  DT  data  from  the  combined  test.  Alternatively,  the 
operational  tester  may  agree  to  write  a  combined  test  report  with 
the  development  tester  and  the  TER  may  contain  the  evaluation 
with  the  combined  DT/OT  report  included  as  an  appendix. 

b.  When  the  DT  and  the  OT  are  combined  into  a  single  test 
which  substitutes  for  EUTE,  LUT,  or  abbreviated  FOT,  an  OA  or  EOA 
is  written  on  the  operational  portions  of  the  test.  lOE 
assessment  also  includes  the  DT  data  from  the  combined  test. 
Alternatively,  the  operational  tester  may  agree  to  write  a 
combined  test  report  with  the  development  tester  and  the  OA  or 
EOA  may  contain  the  assessment  with  the  combined  DT/OT  report 
included  as  an  appendix. 

c.  When  the  DT  and  the  OT  are  combined  into  a  single  test 
which  substitutes  for  any  user  test  of  an  AE  system,  a  TER(AE)  is 
written  on  operational  portions  of  the  test.  Analytic 
conclusions  may  include  DT  data  where  needed  to  address  those  OT 
issues  examined  only  in  DT  portions  of  the  combined  test . 
Alternatively,  the  operational  tester  may  write  a  combined  test 
report  with  the  development  tester  which  contains  the  OT  analytic 
conclusions  as  an  appendix. 


Section  X 
Milestones 


6-34.  Reporting  milestones 

All  milestones  are  to  be  interpreted  as  "not  to  exceed"  dates  for 
planning.  Testers  and  evaluators  will  make  every  effort  to 
complete  reports  and  assessments  as  quickly  as  possible. 

6-35.  Milestone  schedule  for  completion  of  combined  TER,  OA,  and 
EOA  (See  figure  6-6.) 

(INSERT  FIGURE  6-6) 

6-36.  Milestone  schedule  for  completion  of  evaluator-only  OA  and 
EOA  (See  figure  6-7.) 

(INSERT  FIGURE  6-7) 

6-37.  Milestone  schedule  for  completion  of  TER(AE)  and  AOA 
(See  figure  6-8.) 
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(INSERT  FIGURE  6-8) 

6-38.  Milestone  schedule  for  completion  of  TR  (See  figure  6-9.) 

(INSERT  FIGURE  6-9) 


Section  XI 

Methodology  and  Procedures 


6-39.  Report  writing 

a.  Whether  the  report  format  is  a  TER,  TER(AE) ,  OA/EOA,  or 
TR,  test  planning  requirements  for  test  and  evaluation  will  not 
change.  The  only  significant  change  to  result  from  the  TER 
concept  is  a  joint  approach  procedure  by  the  tester  and  evaluator 
for  preparation  of  reports  for  full-evaluate  systems. 

b.  The  responsibilities  of  the  tester  and  the  evaluator 
during  report  preparation  are  described  in  the  following 
paragraphs.  Detailed  guidance  for  report  preparation  is  covered 
in  figure  6-11  at  the  end  of  the  chapter. 

6-40.  TER,  OA,  or  EOA  for  full  evaluate  systems 
This  report  is  the  most  comprehensive  and  important  document 
prepared  by  OPTEC.  With  joint  authors,  it  requires  intensive 
coordination  between  the  tester  and  evaluator.  In  the 
preparation  of  these  reports,  the  following  procedures  will 
apply; 


a.  During  the  development  of  chapters  1  and  2  of  the  TER,  the 
OT  test  team  will  have  the  lead  with  evaluator  input  and 
assistance.  The  evaluator  and  his  analyst  will  work  closely  with 
the  test  team  and  will  normally  be  collocated  with  the  test  team. 

b.  Chapters  1  and  2  are  basically  the  responsibility  of  the 
tester  with  necessary  input  from  the  evaluator.  The  tester  will 
complete  and  publish  the  two  chapters  and  necessary  appendixes  as 
the  PTER-TDDB  or  POA-TDDB.  (See  figure  10-8  for  format.)  These 
two  chapters  will  constitute  the  preliminary  report,  will  reflect 
the  test  results,  and  will  provide  other  DA  and  DOD  agencies 
valuable  information  as  quickly  as  possible. 

c.  During  the  development  of  chapters  3  and  4  of  the  report, 
the  evaluation  system  task  force  will  have  the  lead  with  tester 
input  and  assistance  as  a  member  of  the  system  task  force.  The 
evaluator  will  chair  the  system  task  force.  Chapters  3  and  4  are 
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the  responsibility  of  the  evaluator  with  necessary  input  from  the 
tester. 


d.  Chapter  3  is  prepared  with  significant  contributions  and 
support  from  the  tester  and  from  the  test  analyst.  Only  one  set 
of  level  4  test  data  is  included  in  the  combined  report.  The 
test  officer  and  necessary  test  analysts  must  support  and  provide 
necessary  assistance  to  the  evaluator  during  the  production  of 
the  level  4  and  higher  test  and  evaluation  data. 

e.  Once  these  last  two  chapters  are  finished,  the  evaluator 
(with  tester  input)  completes  the  executive  summary  and  the 
complete  report  is  published. 

f.  The  evaluator  writes  an  OA  or  EOA  for  full-evaluate 
systems  with  no  user  test.  This  OA  or  EOA  normally  has  little, 
if  any,  tester  input. 

6-41.  TER(AE) 

The  tester  will  complete  and  publish  all  four  chapters  of  the  TER 
for  an  AE  system.  The  evaluator  will  complete  the  assessment  in 
an  AOA.  Coordination  is  required  to  assure  the  TER(AE)  and  AOA 
are  properly  "married  up"  with  each  other. 

6-42.  TR  after  FDTE,  CEP,  or  CT 

A  TR  is  written  by  the  tester  and  normally  has  little,  if  any, 
lOE  input. 

6-43.  AOA 

AOA  for  full-evaluate  systems  and  those  AE  systems  where  no 
tester  TER  is  or  will  be  available  to  support  the  decision  point 
are  completed  by  the  evaluator  to  meet  the  suspense  to  provide 
input  to  the  decision  point  and  do  not  have  a  detailed  milestone 
schedule. 

6-44.  Strawman  reports 

a.  Prior  to  start  of  test,  both  the  evaluator  and  the  tester 
should  be  performing  report  related  activities.  In  the  final 
stages  of  planning,  the  tester  and  evaluator  should  be  concerned 
with  ensuring  that  the  data  base  structure  is  developed,  is 
adequate  to  support  the  reports,  and  is  consistent  with  the  TEP. 

b.  As  the  planning  phase  is  being  completed,  the  tester  and 
evaluator  will  begin  development  of  a  strawman  report  based  on 
the  TEP  and  expected  results  from  the  test.  The  strawman  report 
outlines  the  information  to  be  presented  to  ensure  that  voids  or 
gaps  do  not  exist  in  the  TEP.  The  strawman  report  will  include 
all  the  information  that  is  available  prior  to  the  test. 
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c.  For  each  issue,  the  tester  and  evaluator  will  determine 
the  best  method  for  presenting  the  data  to  answer  the  MOE/MOP. 

If  tabular  data  are  to  be  presented,  they  will  show  how  the  data 
will  be  set  up  in  the  table.  If  data  from  questionnaires  are  to 
be  used,  text  can  be  used  with  blank  spaces  for  the  distribution 
of  questionnaire  responses.  They  prepare  an  outline  of  the 
information  to  be  presented  to  decision  makers  to  ensure  that  all 
required  data  to  fully  evaluate  the  system  will  be  obtained  from 
the  operational  test  or  from  or  other  sources. 

d.  The  evaluator  will  use  the  strawman  report  to  outline  how 
he  will  combine  data  from  all  available  sources  into  the  report. 
Of  primary  importance  to  the  tester  are  the  data  displays  of  test 
data  required  by  the  evaluator.  The  outline  for  a  strawman 
report  is  the  same  as  the  format  for  the  final  report  and  should 
be  consistent  with  the  information  provided  in  the  TEP. 


Section  XII 
Offices  of  Record 


6-45.  Office  of  record  for  reports  and  assessments 
The  office  of  record  for  a  T&E  reporting  document  will  be  either 
the  evaluator  or  the  tester.  Offices  of  record  responsibilities 
are  assigned  as  follows; 

a.  Full  evaluate  systems  after  EUTE,  LUT,  lOT,  or  FOT.  The 
tester  is  the  office  of  record  for  these  reports  until  approval 
of  the  preliminary  report.  After  approval  of  the  preliminary 
report,  the  evaluator  becomes  the  office  of  record. 

b.  Abbreviated  evaluate  systems  after  EUTE,  LUT,  lOT,  or  FOT. 
The  tester  is  always  the  office  of  record  for  TER(AE) . 

c.  Full-evaluate  systems  without  EUTE,  LUT,  lOT,  or  FOT.  The 
evaluator  is  always  the  office  of  record  for  all  reports. 

d.  TR  for  FDTE,  CEP,  and  CT.  The  tester  is  always  the  office 
of  record  for  all  TR. 

e.  AOA.  The  evaluator  is  always  the  office  of  record  for  all 
AOA. 

6-46.  Office  of  record  responsibilities 

The  office  of  record  is  responsible  for  the  following; 

a.  Maintenance  of  configuration  control  on  the  document  as  it 
evolves  to  include  maintenance  of  the  "most  current"  version. 
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b.  Incorporation  of  all  changes  regardless  of  the  source  of 
the  changes. 

c.  Maintenance  of  the  approval  sheet  original  copy. 

d.  Printing  and  reproduction  of  copies  for  distribution. 

e.  Submission  of  documents  to  DTIC  and  other  data  bases. 

f.  Maintenance  of  the  record  copy  after  approval. 

g.  All  editorial  services. 

6-47.  Document  and  data  retention 

In  order  to  have  a  complete  historical  file  on  all  tests 
conducted,  each  agency  must  retain  all  documents  for  a  period  of 
time.  Figure  6-10  provides  retention  guidance. 

(INSERT  FIGURE  6-10) 


Section  XIII 
Reporting  Standards 


6-48.  General 

A  report  should  always  be  able  to  stand  the  test  of  clarity  and 
conciseness.  Two  general  characteristics  mark  effective  writing. 
First,  it  has  something  to  say;  and  second,  it  says  only  what  the 
writer  intends  to  say.  It  must  be  free  from  factual  and 
mechanical  errors  and  should  present  only  essential  facts,  free 
from  bias  and  distortion.  Using  a  simple  style  which  avoids 
pretentious  words  and  phrases  will  improve  clarity.  The  reader 
must  be  certain  of  the  writer's  exact  meaning;  therefore,  writing 
must  be  clear  and  free  from  misinterpretation. 

6-49.  Use  of  trade  names  and  code  sheets 

Reports  will  not  advertise  products  or  contain  material  which 
implies  that  the  government  endorses  or  favors  products  of 
commercial  organizations.  Products  will  be  identified  by  either 
generic  names,  standard  Army  nomenclature,  or  specifications. 

When  it  is  essential  that  trade  names  or  manufacturer's  names  be 
used,  the  writer  should  contact  an  editor  to  obtain  the  correct 
procedures. 

6-50.  Draft  report  disclaimer 

Draft  reports  will  carry  the  following  disclaimer:  "The  contents 
of  this  document  reflect  the  views  of  (test  director, 
installation,  location)  and  are  not  to  be  construed  as  the 
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official  position  of  the  Commander,  TEXCOM;  the  Commander,  OPTEC; 
or  the  Department  of  the  Army.” 

6-51.  Classification  markings 

The  materiel  developer  provides  a  system  classification  guide  to 
be  used  in  the  preparation  of  any  test  documents.  If  data 
obtained  during  testing  are  classified,  the  report  has  to  have 
proper  markings.  Document  classifications  are  determined  by  the 
test  officer  and  the  evaluator  at  the  time  of  writing  the  report. 
AR  380-5  provides  classification  guidance. 

6-52.  Tables,  charts,  and  figures 

These  should  be  understandable  even  if  they  are  not  referenced  to 
their  corresponding  paragraphs,  i.e.,  if  they  are  separated  from 
the  report. 

6-53.  Units,  labels,  titles,  and  terms 

These  are  to  be  clear  and  abbreviations  explained  in  footnotes. 
Use  of  these  items  is  to  be  consistent  throughout  the  report. 

6-54 .  Data  explanations 

If  no  data  were  available  for  a  particular  set  of  conditions 
within  a  display  the  notation  ” — ”  is  to  be  used  (not  a  blank 
space)  and  the  reason  why  explained  in  a  footnote.  Definitions 
necessary  for  a  data  presentation  to  stand  alone  are  generally  to 
be  given  in  footnotes.  Explanations  for  unusual  data  points  are 
normally  to  appear  in  footnotes. 

6-55.  Small  data  sets 

When  a  small  data  set  is  small  (listed  on  one  page  or  less),  the 
entire  data  set  is  to  be  given,  not  summarized. 

6-56.  Summary  statistics 

When  summary  statistics  are  computed  for  any  data  set,  explicit 
references  to  the  data  base  are  to  be  given  in  enough  detail  that 
the  presentation  can  be  recreated  from  the  data  base.  Any 
summary  is  to  include  the  following  statistics  as  a  minimum: 
sample  size,  at  least  one  measure  of  central  tendency  (e.g., 
mean,  median)  and  at  least  one  measure  of  variability  (e.g., 
standard  deviation,  range) . 

6-57.  OTE  reporting  definitions 

The  following  definitions  are  important  to  preclude  any  confusion 
prior  to  writing  the  report. 

a.  Test  dates. 

(1)  Test  resource  date  (R-date) . 
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(2)  Test  start  date  (T-date) .  T-date  is  defined  as  the 
date  on  which  data  collection  for  record  begins.  Pretest 
training  and  pilot  test  activities  are  accomplished  prior  to  T- 
date. 


(3)  Test  end  date  (E-date).  E-date  is  defined  as  the  date 
on  which  data  collection  for  record  completes.  Supporting  assets 
are  normally  released  at  or  shortly  after  E-date. 

(4)  Test  completion  date  (C-Date) .  C-date  is  defined  as 
the  date  the  validated  and  authenticated  test  data  base  is 
completed.  Normally,  C-date  will  be  not  later  than  twenty 
calendar  days  after  the  last  test  event. 

b.  Finding.  A  finding  is  a  result  which  is  limited  to  fact 
and  is  derived  from  test  data.  These  data  are  the  results  of 
computations  or  other  mathematical  operations  performed,  or  they 
are  decisions  based  on  the  data  as  preset  conditions  or  criteria. 

c.  Conclusion.  A  conclusion  is  an  interpretive  evaluation  of 
findings  based  solely  on  data  collected  during  the  test. 
Conclusions  will  not  include  any  suitability  type  statement  such 
as  suitability  or  acceptability  for  production,  deployment,  or 
continued  development  of  material. 

d.  Recommendation.  Recommendations  are  based  solely  on 
findings  and  conclusions. 

e.  Observation.  Observations  by  test  directorate  personnel 
or  observers  are  opinions  that  are  part  of  test  conduct  and 
documented  in  chapter  2  and  are  not  part  of  the  data  collection 
process. 

(INSERT  FIGURE  6-11) 
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AND  EVALUATION  REPORTS-TEST  DESCRIPTION  AND 

DATA  BASE 

(PTER-TDDB)  AND  PRELIMINARY  OPERATIONAL 

ASSESSMENT- 

•TEST  DESCRIPTION  AND  DATA  BASE  (POA-TDDB) 

FOR 

FULL-EVALUATE  SYSTEMS  (CONTINUED) 

APPENDIXES. 

APPENDIX  A. 

DATA  BASE  STRUCTURE  AND  DATA  BASE. 

APPENDIX  B. 

SUPPLEMENTAL  USER  TEST  DATA  (Optional) . 

APPENDIX  C. 

RAM  DATA,  COMPUTATIONS,  AND  SCORING 

CONFERENCE  MINUTES. 

APPENDIX  D. 

OTHER  DATA  SOURCES  (Leave  blank) . 

APPENDIX  E. 

SUPPLEMENTAL  ANALYSIS  (Leave  blank) . 

APPENDIX  F. 

SCENARIO  (Optional) . 

APPENDIX  G. 

SYSTEM  DESCRIPTION  (Optional) . 

APPENDIX  H . 

THREAT  (Optional) . 

APPENDIX  I. 

TRAINING  OF  PLAYER  PERSONNEL  (Optional) . 

APPENDIX  J. 

OBSERVATIONS  BY  TEST  TEAM  PERSONNEL  (Leave 
blank) . 

APPENDIX  K. 

COMMENTS  FROM  UNIT  COMMANDERS  (Leave  blank) . 

(through) 

APPENDIX  X. 

AUTHORS  AND  SUPPORTING  PERSONNEL. 

APPENDIX  Y. 

GLOSSARY,  ACRONYMS,  AND  ABBREVIATIONS. 

APPENDIX  Z. 

DISTRIBUTION. 

/  Figure  6-3.  Prescribed  format  for  PTER-TDDB  and  POA-TDDB  for 

V  full-evaluate  systems  (cont) 


Ip- 2.n 


CSTE-EXX 


FORMAT  FOR  FORWARDING  MEMORANDA 
(LETTERHEAD) 

XX  March  199X 


MEMORANDUM  FOR  Appropriate  MADP/MAISRC/IPR  Decision  Authority 

or  Materiel  Release  Authority 


SUBJECT:  Test 

DECISION)  for 
AOA) 


and  Evaluation  Report  (TER)  for  the  (MILESTONE/ 
the  (SYSTEM)  (or  OA,  EOA,  AOA  plus  TER  (AE) , 


1.  Reference.  List  subject  document (s)  and  document 
riqueSting  input  for  the  MADP,  MAISRC,  or  materiel  release 

decision. 


2.  Subject  documents  are  enclosed  for  the  consideration  of 
the  (MADP,  MAISRC,  or  materiel  release  decision) . 

3.  OPTEC  MADP  Position,  MAISRC  Position,  or  Materiel  Release 
Position.  The  OPTEC  position  prepared  by  OEC. 


(full  evaluate) 


(abbreviated  evaluate) 


WILLIE  J.  TOPDOG 
Maj  Gen,  USA 
Commanding 

(or) 

LORENZO  P.  OVERVUE 
COL,  AG 
Commanding 


Enclosures 

1.  TER  (or  OA/EOA) 

2.  AOA  (if  applicable) 

3.  Other  enclosures  are  not 
necessary . 


required,  but  may  be  added  if 


Format  for  forwarding  memoranda 


Figure  6-4 . 


FORMAT  FOR  ABBREVIATED  OPERATIONAL  ASSESSMENTS  (AOA) 

(LETTERHEAD) 

CSTE-EXX  XX  March  199X 


MEMORANDUM  FOR  Commander,  US  Army  Operational  Test  and 

Evaluation  Command,  Park  Center  IV, 

4501  Ford  Avenue,  Alexandria,  VA  22302-1458 

SUBJECT:  Abbreviated  Operational  Assessment  for  the 
(MILESTONE/DECISION)  for  the  (SYSTEM) 


1.  INTRODUCTION:  This  paragraph  may  be  in  narrative  form  and 
need  not  be  broken  into  subparagraphs. 

a.  Purpose.  A  brief  description  of  the  reason (s)  for 
writing  the  AOA  to  include  identification  of  the 
system/program  and  the  MADP,  MAISRC,  or  materiel  release 
decision  supported  by  the  AOA. 

b.  Scope.  A  brief  description  of  the  scope  of  the 
assessment  to  include  mention  of  the  data  sources  considered 
in  the  assessment. 

c.  Evaluation  Limitations.  A  brief  description  of  the 
limitations  of  the  assessment  inherent  in  the  use  of  the  AOA 
format  and  available  data  sources. 

2.  DATA  DESCRIPTION.  A  brief  description  of  each  of  the  data 
sources  listed  in  the  scope  and  considered  in  the  assessment. 
Each  data  source  is  identified  in  a  separate  subparagraph. 

Data  sources  may  include  user  tests,  technical  tests, 
contractor  tests,  market  surveys  and  investigations,  modeling 
and  simulation,  studies  and  analyses,  as  well  as 
programmatics . 

3.  ASSESSMENT  RESULTS.  This  paragraph  may  be  in  narrative 
form  and  need  not  be  broken  into  subparagraphs.  This 
paragraph  provides  a  discussion  of  the  evaluator's  assessment 
of  actual  or  potential  operational  effectiveness  and 
suitability  of  the  system  based  on  any  and  all  available  data. 
When  the  AOA  addresses  a  TER  written  by  the  tester  on  an 
abbreviated  evaluate  system,  this  paragraph  must  address  the 
TER  to  include  an  evaluator  assessment  of  the  consistency  of 
the  data,  completeness  of  test  coverage,  and  validity  of 
conclusions  drawn  (i.e..  Does  the  data  logically  support  the 
evaluative  conclusions?) . 


Figure  6-5.  Format  for  AOA 


FORMAT  FOR  ABBREVIATED  OPERATIONAL  ASSESSMENTS  (AOA) 

CSTE-EXX  _  ,  4.W 

SUBJECT;  Abbreviated  Operational  Assessment  for  the 

(MILESTONE/  DECISION)  for  the  (SYSTEM) 

4.  CONCLUSIONS  AND  RECOMMENDATIONS.  This  paragraph  may  be  in 
narrative  form  and  need  not  be  broken  into  separate 
subparagraphs . 

a.  Overall  Conclusions.  The  conclusions  of  the  evaluator. 

b.  Recommendations.  Recommendations  for  future  testing  of 
the  system  or  for  modifications  to  the  system  or  to  the 
program  to  enhance  operational  effectiveness  or  suitability. 


LORENZO  P.  OVERVUE 
COL,  AG 
Commanding 

Enclosures  ^ 

Enclosures  are  not  required,  but  may  be  added  if  necessary. 

NOTES  ON  USE  OF  AOA;  An  AOA  is  written  in  the  form  of  a 
memorandum  from  Commander,  OEC  to  Commander,  OPTEC.  A 
separate  memorandum  forwards  the  AOA  (and  the  TER  (AE) ,  if 
applicable)  to  the  appropriate  MADP/MAISRC  decision  authority 
or  to  the  materiel  release. authority.  AOA  are  normally  two 
pages  or  less.  Use  the  AOA  for  a  full  evaluate  system  only  to 
report  an  OPTEC  position  for  a  Milestone  I  decision  where  no 
user  test  has  taken  place  or  to  report  an  OPTEC  position  for  a 
relgase  or  other  non^milestone  decision  where  an 
existing  TER  or  OA  exists  to  report  previous  testing  and  a 
milestone  decision.  Use  the  AOA  for  any  and  all  abbreviated 
evaluate  systems  as  follows; 

a.  When  the  tester  writes  a  TER  to  report  the  results  of 
an  EUTE,  LUT,  lOT,  or  FOT  for  an  abbreviated  evaluate  system, 
the  evaluator  prepares  an  AOA  to  be  forwarded  with  the  TER  to 
the  decision  authority. 

b.  When  there  is  no  user  test  to  support  a  decision,  the 
evaluator  prepares  an  AOA  to  report  the  OPTEC  position  to  the 
milestone  decision  body  or  to  the  materiel  release  authority. 


Figure  6-5.  Format  for  AOA  (cont) 


MILESTONE  DATES  FOR  TEST  AND  EVALUATION  REPORTS 
(TER) ,  OPERATIONAL  ASSESSMENTS  (OA) ,  OR  EARLY 
OPERATIONAL  ASSESSMENTS  (EOA)  FOR  FULL-EVALUATE 

SYSTEMS  WITH  OPERATIONAL  TESTING 

Note:  TER  shall  be  construed  to  mean  TER,  OA,  or  EOA 
shall  be  construed  to  mean  PTER-TDDB  or  POA-TDDB. 

PTER 

ACTION 

RESPONSIBILITY 

DATE 

1.  Last  day  of  record  trial 
(i.e.,  last  test  event) 

Test  Officer 

C-20 

2.  Finish  validation  of  test 
data  base  (C-date) 

Test  Officer 

C+00 

3.  Coordination  draft  of  PTER 
provided  to  tester  staff  elements 
(and  to  evaluator  staff  elements) 
for  internal  review  (this  is  the 
complete  draft  PTER  for  staffing) 

Test  Officer 

C+10 

4.  Tester  staff  element  comments 
on  coordination  draft  of  PTER 
returned  to  test  officer 

Tester  HQ 

C+17 

5.  Final  draft  of  PTER  provided 
to  Tester  staff  for  review  and 
command  approval 

Test  Officer 

C+25 

6.  PTER  approved  by  tester 
commander 

Tester  HQ 

C+27 

7 .  Test  and  evaluation  emerging 
results  brief  to  eval  commander 

Evaluator 

C+30 

8.  Normal  release  of  test  team 
(other  than  test  officer,  test 
analyst  and  required  data  management 
personnel  to  eva]  STF) 

Test  Officer 

C+30 

9.  Responsibility  for  office  of 
record  for  TER  preparation  and 
approval  transfers  from  tester  to 
evaluator 

Evaluator 

C+30 

Figure  6-6.  Milestone  dates  for  TER,  OA,  and  EOA  for 
full-evaluate  systems  with  operational  testing 


ACTION 


RESPONSIBILITY  DATE 


10.  PTER  published  and 
distributed 

Tester  HQ 

C+30 

) 

11.  Draft  executive  summary 
of  the  TER  (OA/EOA)  prepared 
and  staffed 

Evaluator 
w/Test  Officer 

C+35 

12.  Draft  executive  summary 
briefed  to  evaluation 
commander 

Evaluator 
w/Test  Officer 

C+44 

13.  Submit  draft  executive 
summary  of  TER  to  OPTEC 
command  review  board  process 

Evaluator 

C+44 

14.  Coordination  draft  TER 
(OA/EOA)  completed  and  compiled 
for  distribution 

Evaluator 
w/Test  Officer 

C+49 

15.  Begin  staffing  of  coordination 
draft  TER  with  tester  staff 
and  evaluator  staff  elements 

Evaluator 
w/Test  Officer 

C+50 

16.  Provide  coordination  draft  of 
TER  to  DOTE  through  DUSA(OR) 

Evaluator 

C+50 

\ 

) 

17.  Staff  comments  on  coordination 
draft  TER  to  evaluator  and  test 
officer 

Evaluator  HQ 
and  Tester  HQ 

C+60 

18.  Command  Review  Board  of  draft 
EXSUM  complete 

OPTEC  HQ 

C+60 

19.  Executive  summary  of  TER 
distribution  to  appropriate  review 
agencies  (copy  to  MDR  body) 

OPTEC  HQ 

C+60 

20.  TER  results  briefing  to  eval¬ 
uation  commander  (may  combine  with 
tester  briefing) 

Evaluator 

C+64 

21.  TER  results  briefing  to  tester 
commander  (may  combine  with 
evaluator  commander  briefing) 

Test  Officer 

C+64 

1'  - - - 

Figure  6-6.  Milestone  dates  for  TER,  OA,  and  EOA  for 
full-evaluate  systems  with  operational  testing  (cont) 


ACTION 

RESPONSIBILITY 

DATE 

22.  Complete  TER  revision  and 
submit  final  draft  TER  to  evalua¬ 
tion  and  tester  commanders  for 
approval 

Evaluator 
w/Test  Officer 

C+69 

23.  TER  approved  by  evaluation 
commander  and  tester  commander 
(final  version  of  TER) 

Evaluation  HQ 
and  Tester  HQ 

C+79 

24.  Submit  final  TER  to  OPTEC 
for  review  and  approval 

Evaluator 

C+80 

25.  Start  OPTEC  Command  Review 
Board  process  on  final  draft  TER 

OPTEC  HQ 

C+80 

26.  Final  draft  TER  to  MDR  body 
for  prebrief 

Evaluator 

C+80 

27.  Final  draft  TER  to  DOTE 
through  DUSA(OR) 

Evaluator 

C+80 

28.  Prebrief  of  results  to  MDR 

Evaluator 

C+80 

29.  OPTEC  Command  Review  Board  on 
TER  complete 

OPTEC  HQ 

C+89 

30.  MDR  (ASARC,  MAISRC,  or  IPR) 
complete 

Decision 

Makers 

C+90 

31.  OPTEC  approves  TER 

OPTEC  HQ 

C+93 

32.  Prebrief  DOD  Committee  using 
approved  executive  summary  and  TER 

Evaluator 

C+100 

33.  Approved  TER  published  and 
distributed 

Evaluation  HQ 

C+104 

34.  Defense  Acquisition  Board  ' 

(DAB)  held 

DOD 

C+130 

Figure  6-6.  Milestone  dates  for  TER,  OA,  and  EOA  for 
full-evaluate  systems  with  operational  testing  (cont) 


MILESTONE  DATES  FOR  OPERATIONAL  ASSESSMENTS  (OA) 
OR  EARLY  OPERATIONAL  ASSESSMENTS  (EOA)  FOR  FULL- 
EVALUATE  SYSTEMS  WITH  NO  OPERATIONAL  TESTING 


NOTE:  OA  shall  be  construed  to  mean  OA  or  EOA. 


ACTION 

RESPONSIBILITY 

DATE 

1.  Date  of  cutoff  of  data  from 
sources  (DT,  Cont  Test,  Mod/ Sim) . 

Evaluator 

C+00 

2.  Test  and  evaluation  emerging 
results  brief  to  the  evaluation 

Evaluator 

C+10 

commander 

3.  Draft  executive  summary  of 
the  OA  prepared  and  staffed 

Evaluator 

C+19 

4.  Draft  executive  summary 
briefed  to  evaluation  commander 

Evaluator 

C+29 

5.  Submit  draft  executive 
summary  of  OA  to  command  review 
board  process 

Evaluator 

C+30 

6.  Coordination  draft  OA 
completed  and  compiled  for 
distribution 

Evaluator 

C+34 

7.  Begin  staffing  of  coordination 
draft  OA  with  evaluator  staff 

Evaluator 

C+34 

elements 

8.  Provide  coordination  draft  of 

OA  to  DOTE  through  DUSA(OR) 

Evaluator 

C+34 

9.  Staff  comments  on  coordination 
draft  OA  to  evaluator 

Evaluator  HQ 

C+44 

10.  Command  Review  Board  of  draft 

OPTEC  HQ 

C+44 

executive  summary  complete 

11.  The  executive  summary  of  OA  OPTEC  HQ  C+44 

approved  for  distribution  to 
appropriate  review  agencies 
(copy  to  MDR  body) 


Figure  6-7.  Milestone  dates  for  OA  or  EOA  for  full-evaluate 
systems  with  no  operational  testing 


ACTION 

RESPONSIBILITY 

DATE 

12.  OA  results  briefing  to 
evaluation  commander 

Evaluator 

C+48 

13.  Complete  OA  revision  and 
submit  final  draft  to  evaluation 
commander  for  approval 

Evaluator 

C+54 

14.  OA  approved  by  evaluation 
commander  (final  version  of  OA) 

Evaluation  HQ 

C+54 

15.  Submit  final  OA  to  OPTEC 
for  review  and  approval 

Evaluator 

C+55 

16.  Start  OPTEC  Command  Review 
Board  process  on  final  draft  OA 

OPTEC  HQ 

C+55 

17.  Final  draft  OA  to  MDR  body 
for  prebrief 

Evaluator 

C+55 

18.  Final  draft  OA  to  DOTE 
through  DUSA(OR) 

Evaluator 

C+55 

19.  Prebrief  of  results  to  MDR 

Evaluator 

C+55 

20.  OPTEC  Command  Review  Board  on 
OA  complete 

OPTEC  HQ 

C+64 

21.  MDR  (ASARC,  MAISRC,  or  IPR) 
complete 

Decision 

Makers 

C+65 

22.  OPTEC  approves  OA 

OPTEC  HQ 

C+67 

23.  Approved  OA  published  and 
distributed 

Evaluation  HQ 

C+72 

Figure  6-7.  Milestone  dates  fpr  OA  or  EGA  for  full-evaluate 
systems  with  no  operational  testing  (cont) 


MILESTONE  DATES  FOR  TEST  AND  EVALUATION  REPORTS  (TER) 
AND  ABBREVIATED  OPERATIONAL  ASSESSMENTS  (AOA)  FOR 
ABBREVIATED  EVALUATE  SYSTEMS  WITH  OPERATIONAL  TESTING 


ACTION 

RESPONSIBILITY 

DATE 

1.  Last  day  of  record  trial 
(i.e.,  last  test  event) 

Test  Officer 

C-20 

2.  Finish  validation  of  test 
data  base  (C-Date) 

Test  Officer 

C+00 

3.  Coordination  draft  TER 
completed  and  compiled  for 
distribution 

Test  Officer 

C+39 

4.  Begin  staffing  of  coordination 
draft  TER  with  tester  staff 
and  evaluator  staff  elements 

Test  Officer 

C+39 

5.  Staff  comments  on  coordination 
draft  TER  to  test  officer 
officer 

Evaluator  HQ 
and  Tester  HQ 

C+49 

6.  Begin  writing  AOA 

Evaluator 

C+50 

7.  Revision  of  TER  to  address 
agency  comments  completed 

Test  Officer 

C+59 

8.  Final  draft  TER  submitted 
to  tester  HQ  for  typesetting 

Test  Officer 

C+61 

9.  Copy  of  camera-ready  TER 
returned  to  tester  for  final  review 

Tester  HQ 

C+71 

10.  Camera-ready  TER  and  any  final 
comments  returned  to  Tester  HQ  for 
final  processing 

Test  Officer 

C+74 

11.  TER/ AOA  results  briefing  to 
evaluation  commander  (may  combine 
with  tester  briefing) 

Evaluator 

C+75 

12.  TER  results  briefing  to  tester 
commander  (may  combine  with 
evaluator  briefing) 

Test  Officer 

C+75 

Figure  6-8.  Milestone  dates  for  TER  and  AOA  for  abbreviated 
evaluate  systems  with  operational  testing 


^,-36 


ACTION 

RESPONSIBILITY 

DATE 

13.  Complete  TER  revision  and 
submit  final  draft  TER  to  tester 
commander  for  approval 

Test  Officer 

C+77 

14.  TER  approved  by  tester 
commander  (final  version  of  TER) 

Tester  HQ 

C+79 

15.  Complete  AOA  and  submit  to 
evaluation  commander  for  approval 

Evaluator 

C+80 

16.  Approved  TER  published  and 
distributed 

Tester  HQ 

C+85 

17.  AOA  approved  by  evaluation 
commander 

Evaluator  HQ 

C+85 

18.  MDR  (IPR) 

Decision  Makers 

C+115 

Figure  6-8.  Milestone  dates  for  TER  and  AOA  for  abbreviated 
evaluate  systems  with  operational  testing  (cont) 


MILESTONE  DATES  FOR  TEST  REPORTS  (TR)  FOR 
FDTE  AND  FOR  CEP  AND  CT  (IN  PARENTHESES) 

ACTION 

RESPONSIBILITY 

DATE 

1.  Last  day  of  record  trial 
(i.e.,  last  test  event) 

Test  Officer 

C-20 

2.  Finish  validation  of  test 
data  base  (C-Date) 

Test  Officer 

C+00 

3.  Coordination  draft  TR  completed 
and  compiled  for  distribution 

Test  Officer 

C+39 

(C+19) 

4.  Begin  staffing  of  coordination 
draft  TR  with  tester  staff  and 
and  combat  or  training  developer 
staff  elements 

Test  Officer 

C+39 

(C+19) 

5.  Staff  comments  on  coordination 
draft  TR  to  test  officer 

Tester  HQ 
and  CD/TD 

C+49 

(C+27) 

6.  Revision  of  TR  to  address 
agency  comments  completed 

Test  Officer 

C+59 

(C+34) 

7.  Final  draft  TR  submitted 
to  tester  HQ  for  typesetting 

Test  Officer 

C+61 

(C+35) 

8.  Copy  of  camera-ready  TR  returned 
to  tester  for  final  review 

Tester  HQ 

C+71 

(C+45) 

9.  Camera-ready  TR  and  any  final 
coinments  returned  to  Tester  HQ  for 
final  processing 

Test  Officer 

C+74 

(C+47) 

10.  TR  results  briefing  to  tester 
commander  (may  include  CD/TD  repre¬ 
sentatives)  (not  required  for  CEP/CT) 

Test  Officer 

C+75 

(NONE) 

11.  Complete  TR  revision  and  submit 
final  draft  TR  to  tester  commander 
for  approval 

Test  Officer 

C+77 

(C+49) 

12.  TR  approved  by  tester 
commander  (final  version  of  TR) 

Tester  HQ 

C+79 

(C+50) 

13.  Approved  TR  published  and 
distributed 

Tester  HQ 

C+85 

(C+60) 

Figure  6-9.  Milestone  dates  for  TR  for  FDTE  and  for  CEP  and  CT 


DATA  RETENTION  GUIDANCE 


Data 

Category 

Subcategory 

Retention/ 

Review 

Period* 

Storage/Media 

Accessibility 

D.  Raw 

Data. 

Data  in  its 

original 

form. 

D.l  Exposed  film 
and  original 
video  tapes. 

D.2  Original 
instrumentation 
tapes  or 
printouts . 

D.3  Completed 
data  collection 
sheets  and 
questionnaires . 

D. 4  ... 

Not  stored  if 
Level  2  data 
has  been 
developed 
from  it  and 
is  available; 
otherwise, 
retain  until 
report 
published. 

Retain  in 
original  form 
(off  line) . 

V. 

Audio/Video 
Tape  and 

Film. 

V.l  Labeled  audio 
or  video  tape  or 
film. 

V. 1 . a  ... 

V.l.b  . . . 

V.2  Edited  audio 
or  video  tape  of 
film. 

4  yrs/2  yrs. 

As  recorded 
(off  line) . 
Digital 
optical  media 
should  be 
considered 
for  retention 
of  4  yrs  or 
longer. 

1 

Processed 

and 

Smoothed 

Automated 

Instrumen¬ 

tation 

Data. 

1.1  Position/ 
location. 

1.2  Line  of 
sight. 

1.3  Data  bus. 

1.4  ... 

10  yrs/2  yrs. 

Tape  or 
digital 
optical 
media. 

A. 

Automated 
Test  Data 
Base  of 
Record . 

A.l  Data  base. 

A. 2  Data  base 
dictionary. 

A . 3  Programs 
which  build  data 
base. 

.4  ... 

10  yrs/2  yrs. 

i 

Tape  or 
digital 
optical 
media. 

Figure  6-10.  Data  retention  guidance 


W.  Written 
Level  2  and 
Level  3 

Data. 

W.l  Corrected 
manual  forms. 

W.2  Logs,  tran¬ 
scripts, 
printouts. 

W.3  Reduced 
manual  data. 

W.l  1  yr/2 
yrs. 

W.2  4  yrs/2 
yrs. 

W.3  4  yrs/2 
yrs. 

Paper  (off 
line)  . 

Digital 
optical  media 
should  be 
considered 
for  retention 
of  4  yrs  or 
longer. 

R.  Plans, 
Reports , 
and 

Analysis. 

R.l  Plans, 
reports . 

R.2  Analysis 
programs . 

R. 3  ... 

R.l 

Permanent . 

R.2  3  yrs/2 
yrs. 

— - 1  - » - 

R.l  Digital 

optical 

media. 

(Paper  copy 
of  report  to 
DTIC 

required. ) 

R.2  Tape  or 
digital 
optical 
media. 

*  Minimum  retention  period  and  maximum  period  between  subsequent 
reviews . 


1 


Figure  6-10.  Data  retention  guidance  (cont) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND  DOCUMENTATION  OF 
TEST,  EVALUATION,  AND  ASSESSMENT  REPORTING  DOCUMENTS 

FORMAT 

FRONT  COVER 

APPROVAL  PAGE 

DISCLAIMER  FOR  TRADE  NAMES  (if  required) 

REPORT  DOCUMENTATION  PAGE,  STANDARD  FORM  298 
EXECUTIVE  SUMMARY 

I.  PURPOSE.  Succinctly  state  the  purpose  of  any  evaluation, 
assessment,  or  user  test  reported  in  the  document.  This 
paragraph  is  an  abridgement  of  paragraph  1.1.  in  chapter  1  of 
this  report. 

II.  BACKGROUND.  Summarize  necessary  background  information 
contained  in  paragraph  1.3.  in  chapter  1  of  this  report. 
Include  information  on  previous  tests  that  may  have  an  impact 
on  understanding  the  test,  evaluation,  assessment,  or 
decision. 

III.  SYSTEM  DESCRIPTION  (For  TR  this  paragraph  should  be 
titled  "Description.”)  Summarize  pertinent  system  or  product 
description  information  contained  in  paragraph  1.4.  in  chapter 
1  of  this  report.  Include  only  sufficient  information  to 
identify  the  system  under  discussion  and  to  explain  its  role 
and  its  missions  within  the  Army  branch  functional  areas. 

IV.  TEST  CONDUCT  (For  OA/EOA  Which  do  not  include  user 
testing,  this  paragraph  should  be  titled  "DATA  SOURCES,"  see 
paragraph  b.  below.) 

a.  IV.  TEST  CONDUCT.  Do  not  use  for  OA/EOA  with  no  user 
test.  This  paragraph  covers  the  what,  where,  when,  and  how  of 
the  test.  Provide  specifics  that  show  the  extent  of  the  test, 
as  to  duration  of  the  test,  number  of  trials,  number  and  types 
of  crews,  and  overall  conditions.  Give  the  location  and  dates 
of  the  test  and  who  conducted  it.  Provide  a  brief  description 
of  the  test  execution.  Summarize  details  from  the  scope, 
general  methodology,  and  test  execution  in  chapter  2  of  the 
TER. 
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b.  IV.  DATA  SOURCES.  Use  only  for  OA/EOA  with  no  user 
test.  This  paragraph  covers  the  data  sources  used  to  conduct 
the  assessment  and  prepare  the  OA/EOA.  Summarize  details  from 
chapter  2  of  the  OA/EOA.  Provide  specifics  that  show  the 
extent  of  the  test  (model,  simulation,  study,  etc.)  and  the 
location  and  dates  of  the  effort  and  who  conducted  it. 

V.  TEST  AND  EVALUATION  LIMITATIONS  AND  IMPACT  (For  TER  (AE) 
or  TR  this  paragraph  should  be  titled  "Test  Limitations  and 
Impact";  for  OA/EOA  after  EUTE,  LUT,  or  abbreviated  FOT  use 
the  title  "Test  and  Assessment  Limitations  and  Impact";  and 
for  OA/EOA  with  no  user  test  "Assessment  Limitations  and 
Impact . " ) 

a.  List  both  the  limitations  for  the  conduct  of  the  user 
test  (if  any)  and  the  limitations  on  the  evaluation/assessment 
(except  in  TER  (AE)  and  TR) .  List  all  the  limitations  if  the 
number  is  small.  If  the  number  is  large,  list  the  major 
limitations  in  this  paragraph  and  cover  all  of  the  limitations 
in  paragraph  1.7.  of  the  report. 

b.  Include  a  statement  that  no  limitations  existed,  if 
applicable. 

VI.  OPERATIONAL  EFFECTIVENESS  SUMMARY  (Title  this  paragraph 
"Conclusions"  for  TR  of  FDTE,  CEP,  and  CT.)  Present 
significant  results  and  the  evaluation/assessment/analysis 
summary  that  will  address  that  portion  of  the  operational 
issues  that  contribute  to  operational  effectiveness  of  the 
system  or  product. 

VI. 1.  Issue  Summary.  Copy  each  issue  statement  from 
paragraph  1.5.  in  chapter  1  of  this  report.  This  is  a  summary 
of  chapter  3  of  this  report. 

VI. 1.1.  Issue  1.  This  is  the  first  issue  from  paragraph  1.5. 
of  this  report.  State  the  issue  as  written  in  the  TEP,  then 
provide  a  narrative  as  to  how  the  answers  to  the  issue 
contribute  to  operational  effectiveness  (effectiveness  in  the 
TR)  of  the  system  or  product.  Provide  the  significant  results 
from  all  data  sources  and  the  writer's  assessments  of  the 
issue  based  on  the  results. 


Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 


I 


(p 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND  DOCUMENTATION  OF 
TEST,  EVALUATION,  AND  ASSESSMENT  REPORTING  DOCUMENTS 

(CONTINUED) 

VI. 1.2.  Issue  2.  This  is  the  second  issue  from  paragraph  1.5 
of  this  report. 

(through) 

Vl.l.n.  Issue  n.  This  is  the  n-th  issue  from  paragraph  1.5 
of  the  TER. 

VI. 2.  Overall  Operational  Effectiveness  Assessment  (Title 
this  paragraph  "Overall  Assessment"  for  a  TR.)  This  is  a 
summary  of  paragraph  4.1.  of  the  report. 

VI.  3.  Operational  Effectiveness  Recommendations  (Title  this 
paragraph  "Recommendations"  for  a  TR.)  This  paragraph  is  not 
used  for  TER  (AE) .  This  is  a  summary  of  the  operational 
effectiveness  portion  of  paragraph  4.1.  of  the  report 
(paragraph  4.2.  of  a  TR) . 

.VII .  OPERATIONAL  SUITABILITY  SUMMARY.  This  paragraph  is  not 
used  for  a  TR  of  FDTE,  CEP,  or  CT.  Present  the  major  results 
and  the  evaluation/assessment/analytic  summary  that  will 
address  that  portion  of  the  operational  issues  that  contribute 
to  operational  suitability. 

VII.  1.  Issue  Summary.  Copy  each  issue  statement  from 
paragraph  1.5.  of  this  report.  This  is  a  summary  of  chapter  3 
of  this  report. 

VII. 1.1.  Issue  1.  This  is  the  first  issue  from  paragraph 
1.5.  of  this  report.  State  the  issue  as  written  in  the  TEP, 
then  provide  a  narrative  as  to  how  the  answers  to  the  issue 
contribute  to  operational  suitability  of  the  system.  Provide 
the  significant  results  from  all  data  sources  and  the 
evaluator  assessments  of  the  issue  based  on  the  results. 

VII. 1.2.  Issue  2.  This  is  the  second  issue  from  paragraph 
1.5.  of  this  report. 

(through) 

VII. l.n.  Issue  n.  This  is  the  n-th  issue  from  paragraph  1.5. 
of  this  report. 

VII. 2.  Overall  Operational  Suitability  Assessment.  This  is  a 
summary  of  paragraph  4.2.  of  this  report. 
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VII.  3.  Operational  Suitability  Recommendations.  This  is  a 
summary  of  the  operational  suitability  portion  of  paragraph 
4.3.  of  this  report.  This  paragraph  is  not  used  in  an  TER 
(AE) . 

VIII.  RECOMMENDATIONS  POR  FUTURE  0TB.  This  paragraph  is  not 
used  for  an  TER  (AE)  or  a  TR  for  FDTE,  CEP,  or  CT.  Summarize 
the  recommendations  contained  in  paragraph  4.3.2.  of  the 
report. 

TABLE  OF  CONTENTS 
LIST  OF  TABLES 
LIST  OF  FIGURES 
■LIST  OF  ILLUSTRATIONS 
BODY  OF  REPORT 
CHAPTER  1.  INTRODUCTION. 

a.  The  tester  leads  in  the  preparation  of  this  paragraph 
for  TER  with  input  by  the  evaluator.  The  tester  prepares  this 
chapter  for  TER  (AE)  and  TR.  The  evaluator  prepares  this 
chapter  for  OA/EOA  with  no  user  test. 

b.  This  chapter  provides  necessary  information  to 
understand  the  bases  of  both  the  test  and  of  the  evaluation. 
The  writer  will  extract  and  update  material  as  necessary  from 
chapter  1  of  the  approved  TEP  to  complete  the  paragraphs  for 
this  chapter. 

c.  The  tester  and  evaluator  should  complete  most  of  this 
chapter  when  they  begin  the  strawman  report. 

1.1.  PURPOSE. 

a.  This  paragraph  states  the  purposes  for  conducting  the 
evaluation  or  assessment  of  the  test  as  it  relates  to  the 
program  acquisition  strategy  and  structure.  Identify  MADP 
milestone  or  other  decision  supported  by  the  evaluation  or 
assessment. 
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b.  The  paragraph  should  be  a  succinct  statement  of  the 
purpose  of  the  evaluation  or  assessment  and  the  supporting 
user  test.  Use  the  purpose  statement  from  the  TEP  as  a  guide 
and  update  it  using  the  past  tense. 

1.2.  SCOPE. 

a.  This  paragraph  addresses  the  breadth  of  the  data 
sources  used  to  prepare  this  TER,  OA,  or  EOA.  Provide  an 
overview  of  relevant  models,  analyses,  tests,  equipment,  and 
other  resources  that  together  are  the  basis  for  the  evaluation 
or  assessment.  The  scope  also  addresses  the  depth  of 
evaluation  or  assessment  of  both  operational  effectiveness  and 
suitability. 

b.  For  full-evaluate  systems,  the  scope  of  the  evaluation 
or  assessment  will  include  information  on  all  the  User  Tests, 
Technical  Tests,  modeling  and  simulation,  and  studies  that 
provided  information  for  the  evaluation  or  assessment.  Use 
the  scope  paragraph  from  the  TEP  as  a  guide  and  update  it 
using  the  past  tense.  The  data  source  matrix  in  the  TEP 
provides  the  outline  for  the  scope  of  the  TER.  Describe  the 
test(s)  identified  in  the  system  TEMP  and  TEP  covered  by  this 
report . 

c.  The  EOA  for  a  Milestone  II  will  typically  assess 
specific  system  capabilities  and  potential  for  maturation. 

d.  The  OA  for  a  LRIP  Release  Decision  will  typically 
assess  operational  effectiveness  and  suitability  and  readiness 
for  LRIP  and  lOT. 

e.  The  TER  for  a  Milestone  III  will  typically  evaluate 
operational  effectiveness  and  suitability  and  support  the 
full-scale  production  decision. 

f.  For  AE  systems  and  TR,  use  the  scope  paragraph  in  the 
TEP  as  a  guide  and  update  it  using  the  past  tense.  Combat  and 
training  developers  may  conduct  evaluations  or  assessments  in 
support  of  their  products. 
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1.3.  BACKGROUND.  This  paragraph  includes  the  background  of 
the  system  development  and  the  test  and  evaluation  of  the 
system.  For  TR  for  FDTE,  CEP,  and  CT,  this  paragraph  includes 
the  background  of  the  system  or  combat /training  development 
product  and  the  previous  testing  of  the  system/product. 

1.3.1.  PROGRAM  BACKGROUND.  Use  the  Program  Background 
paragraph  from  the  TEP  as  a  guide  and  update  it  using  the  past 
tense.  Include  an  overview  of  the  program  and  its  acquisition 
strategy  or  the  combat  or  training  development  and  its  planned 
implementation.  State  the  anticipated  use  to  the  Army  and 
deficiencies  that  the  system/product  corrects.  State  the  next 
program  decision  point  (or  CD/TD  equivalent)  supported  by  the 
testing  and  evaluation  and  the  required  decision  from  that 
meeting  (i.e.,  enter  next  acquisition  phase,  low  rate  initial 
production,  full-scale  production,  fielding) .  Identify 
deficiencies  or  suitability  and  effectiveness  problems 
existing  in  similar  systems/products,  as  well  as  the  test  and 
programmatics  used  to  evaluate  those  systems /products. 

1.3.2.  TEST  AND  EVALUATION  BACKGROUND.  This  paragraph 
includes  a  summary  of  all  operational,  technical,  contractor, 
and  other  user  (FDTE,  CEP,  CT)  test  and  evaluation  to  date. 
Include  both  the  scope  and  the  results  of  the  test  and 
evaluation. 

1.3.3.  COEA/OTE  RELATIONSHIP.  Required  for  any  system  for 
which  a  COEA  is  done  and  OTE  is  conducted.  Describe  the 
linkage  between  the  COEA  and  the  results  of  OTE.  The 
description  of  the  linkage  should  explain  how  the  MOEs  and 
MOPS  used  for  OTE  are  consistent  with  the  criteria  in  the 
COEA,  which  in  turn  should  have  MOEs  and  MOPs  consistent  with 
the  operational  requirements  document  (ORD) ,  the  TEMP  and  the 
Acquisition  Program  Baseline  (APB) . 


Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 


DETAILED  GUIDANCE  FOR  THE  DEVELOPMENT  AND  DOCUMENTATION  OF 
TEST,  EVALUATION,  AND  ASSESSMENT  REPORTING  DOCUMENTS 

(CONTINUED) 

1.4.  SYSTEM  DESCRIPTION  (Title  this  paragraph  "Description” 
for  TR  of  FDTE,  CEP,  and  CT.) 

a.  This  paragraph  includes  a  description  of  the  materiel 
or  IMA  system  or  the  combat/training  developments  product.  It 
should  describe  the  system/product  and  its  major  roles, 
missions,  and  components  or  characteristics.  A  description  of 
similarities , and  differences  between  the  system  or  product 
under  test  and  the  objective  system  or  product  under 
development  is  appropriate  here.  Summarize  the  concept  for 
force  structure  and  employment  in  this  paragraph. 

b.  Use  the  description  paragraph  from  the  TEP  as  a  guide 
and  update  it  using  the  past  tense.  If  the  details  are  so 
voluminous  that  continuity  of  the  TER  is  lost,  they  should  be 
summarized  here  and  stated  in  detail  in  appendix  G.  Appendix 
G  is  optional  if  the  details  can  be  adequately  covered  in  this 
paragraph. 

1.5.  OPERATIONAL  ISSUES  AND  CRITERIA  (OIC) . 

a.  This  paragraph  lists  the  operational  issues  and  their 
associated  criteria  used  to  evaluate,  assess  or  analyze  the 
system  and  that  were  the  basis  for  constructing  the 
evaluation,  assessment  or  analytic  plan  and  developing  the 
test  design  for  the  system  or  for  the  combat  or  training 
developments  product.  The  OIC  listed  here  are  those  OIC,  both 
critical  and  additional,  stated  in  the  TEP.  State  the  full 
issue,  scope,  criteria,  and  rationale. 

b.  For  materiel  and  IMA  systems,  the  OIC  will  be  broken 
down  into  COIC  (prepared  by  the  combat  developer  or  functional 
proponent)  and  AOIC  prepared  by  the  evaluator.  The  COIC  and 
the  AOIC  collectively  constitute  the  OIC  for  evaluation, 
assessment,  or  analysis.  For  test  of  combat  or  training 
developments  products,  OIC  are  developed  and  stated  by  the 
combat  developer,  training  developer,  or  functional  proponent, 
as  appropriate. 
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1.5.1.  CRITICAL  OPERATIONAL  ISSUES  AND  CRITERIA  (COIC) . 

a.  This  paragraph  contains  the  approved  COIC  which  are 
directly  extracted  from  the  approved  TEP.  The  critical 
operational  issues  address  key  operational  questions  about  a 
system  which  must  be  addressed  for  the  next  milestone 
decision.  The  emphasis  of  critical  issues  is  on  determination 
of  the  attainment  of  certain  key  performance  levels  and  on 
surfacing  potential  problems  which  could  interfere  with  a 
successful  acquisition. 

b.  Each  COIC  set  is  stated  in  full  to  include  issue, 
scope,  criteria,  and  rationale.  The  evaluator  may  have  added 
additional  evaluator  criteria  to  a  critical  issue.  Criteria 
which  are  a  part  of  the  original  COIC  set  will  always  have 
both  a  measure  (parameter)  and  a  threshold  (value) .  Criteria 
developed  by  the  evaluator  may  have  only  a  measure  (parameter) 
specified. 

c.  Subparagraph  organization  is  as  follows: 

1.5. 1.1.  COIC  1  Issue  statement 

1.5. 1.1.1.  COIC  1  Scope 

1.5. 1.1. 2.  COIC  1  Criteria 

1.5. 1.1. 3.  COIC  1  Rationale 

1.5. 1.2.  COIC  2 

through 

1.5.1.m.  COIC  m 

1.5.2.  ADDITIONAL  OPERATIONAL  ISSUES  AND  CRITERIA  (AOIC) . 

a.  This  paragraph  contains  the  AOIC  developed  by  the 
evaluator  to  complement  and  supplement  the  approved  COIC  to 
completely  address  all  aspects  of  system  operational 
effectiveness  and  suitability.  The  approved  AOIC  are  directly 
extracted  from  the  approved  TEP.  The  AOIC  address  operational 
questions  about  a  system  which  must  be  addressed  for  a 
complete  operational  evaluation  or  assessment  to  be  made  about 
a  system.  The  emphasis  of  AOIC  issues  is  on  determination  of 
the  attainment  of  all  performance  levels  and  on  surfacing 
potential  problems  which  could  affect  the  acquisition. 

b.  Each  AOIC  set  is  stated  in  full  to  include  issue, 
scope,  criteria,  and  rationale.  Criteria  developed  by  the 
evaluator  may  have  only  a  measure  (parameter)  specified. 
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c.  Subparagraph  organization  is  as  follows: 

AOIC  B<fl  Issue  StatSBent 

1.5.2. B-I'1.1.  AOIC  B-l-1  Scope 

1.5.2. B4-1.2.  AOIC  B-fi  Criteria 

1.5.2. B-fl.3.  AOIC  B-fl  Rationale 
1.5.2.B-f2.  AOIC  B-l-2 

through 

l*5*2«n*  AOIC  n 

1.6.  EVALUATION  APPROACH  (Title  this  paragraph  as 
"Assessment  Approach"  in  all  OA/EOA;  "Analytic  Approach"  in 
all  TER  (AE)  and  TR.) 

a.  This  paragraph  summarizes  the  overall  approach  to  the 
evaluation  of  the  system.  Show  the  methodologies  by  which  the 
answers  to  the  issue  questions  were  further  analyzed  and 
consolidated  to  draw  conclusions  on  the  overall  operational 
effectiveness  and  operational  suitability  of  the  system. 

b.  Include  in  this  paragraph  the  overall  evaluation, 
assessment,  or  analytic  approach  as  stated  in  paragraph  2.1  of 
the  TEP.  Include  references  to  the  data  source  matrix  and  the 
purpose  of  other  tests  conducted  in  support  of  the 
evaluation/assessment/analytic  plan  as  applicable. 

1.7.  TEST  AND  EVALUATION  LIMITATIONS  AND  IMPACT  (Title  this 
paragraph  "Test  Limitations  and  Impact"  for  TER  (AE)  and  TR; 
"Test  and  Assessment  Limitations  and  Impact"  for  OA/EOA  after 
EUTE,  LUT,  and  abbrev  FOT;  and  "Assessment  Limitations  and 
Impact"  after  OA/EOA  with  no  user  test.) 

1.7.1.  TEST  LIMITATIONS  AND  IMPACT  (Number  this  paragraph 
1.7  for  TER  (AE)  and  TR.) 

a.  This  paragraph  is  prepared  by  the  tester  and  states 
the  limitations  that  prevented  complete  test  execution, 
impacted  the  collection  of  data,  or  affected  the  data 
collected.  Describe  uncontrolled  variables  not  discussed 
above.  Describe  variables  over  which  the  tester  had  no 
control  (e.g.,  player  morale,  leadership,  weather)  and  the 
impact  of  each  on  the  test. 
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b.  List  all  limitations  that  had  an  impact  on  test  design 
and  test  conduct.  Describe  all  known  limitations  to  the 
adequacy  of  the  test  (e.g.,  duration,  number  of  systems,  test 
unit  availability,  availability  of  interoperable  systems,  and 
instrumentation)  and  their  impact  on  the  operational  test 
measures  of  performance  (OTMOPs) . 

c.  List  any  deviations  from  the  TEP,  why  they  occurred, 
and  the  impact  they  had  on  the  test. 

1.7.2.  EVALUATION/ASSESSMENT  LIMITATIONS  AND  IMPACT  (Number 
this  paragraph  1.7  for  OA/EOA  with  no  user  test.) 

a.  This  paragraph  is  prepared  by  the  evaluator  and  states 
the  known  limitations  of  the  evaluation  and  those  data  sources 
(to  include  the  operational  test)  which  supported  the 
evaluation.  These  limitations  are  to  be  presented  and 
discussed  in  terms  of  their  impact  on  the  evaluation.  This 
will  include  an  explanation  of  why  the  limitations  existed, 
which  COIC/AOIC  were  impacted,  and  what  was  done  to  minimize 
their  impact. 

b.  List  all  limitations  that  had  an  impact  on  the  data 
and,  therefore,  on  the  evaluation.  List  the  impact  on  the 
evaluation  or  assessment  of  any  deviations  from  the  TEP. 

CHAPTER  2.  TEST  DESCRIPTION. 

a.  For  OA/EOA  with  no  user  test,  chapter  2  is  a  one  line 
statement  to  the  effect  that  no  dedicated  user  test  was 
planned  or  conducted  to  support  the  OA/EOA. 

b.  The  basic  format  for  chapter  2  in  all  cases  other  than 
that  mentioned  above  is  as  follows.  The  tester  prepares  all 
of  the  paragraphs  in  this  chapter.  This  paragraph  describes 
the  test  conducted  to  address  the  operational  issues,  OTMOP, 
and  data  requirements  specified  in  the  TEP. 

c.  Use  appendixes  only  as  necessary  where  the  material  is 
too  voluminous  to  include  in  this  paragraph  without  breaking 
the  logical  flow  of  information.  Whenever  using  an  appendix, 
provide  a  brief  summary  of  the  information  within  this 
chapter . 
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d.  For  some  FDTE,  CEP,  and  CT  supporting  combat/training 
development  products,  all  paragraphs  may  not  apply.  If  a 
paragraph  is  not  applicable  to  this  test,  so  state. 

2.1.  TEST  PURPOSE.  State  the  test  purpose  from  the  approved 
outline  test  plan  (OTP)  or  Resume  Sheet  (RS)  for  the  test. 

This  paragraph  contains  the  purpose  of  the  test  and  proposed 
usage  of  the  results.  This  will  be  the  same  as  the  test 
purpose  contained  in  the  TEP. 

2.2.  AUTHORITY  TO  CONDUCT  TEST.  State  the  specific  TEMP  and 
approved  5-year  test  plan  (FYTP)  authorizing  the  conduct  of 
test,  and  reference  the  DA  memorandum  approving  the  current 
FYTP  and  the  number  of  the  approved  OTP  or  RS  for  the  test. 

2.3.  TEST  OVERVIEW. 

a.  Provide  a  general  overview  of  the  test  execution,  to 
include  when,  where,  general  description,  and  test  variables. 
Refer  to  chapter  2  of  the  TEP  where  the  evaluator  described 
the  concepts  guiding  the  design  and  execution  of  the  test. 

b.  Specify  the  test  phases  (e.g.,  pilot  test,  record 
trials,  live  firings,  field  training  exercises  (FTXs) ,  command 
post  exercises  (CPXs) ,  demonstrations)  making  up  the  test  and 
summarize  test  conditions  and  events  making  up  each  phase. 
Describe  use  of  any  baseline  system  in  testing.  Include 
descriptions  of  execution  procedures,  control  procedures,  and 
any  other  information  needed  to  understand  the  matrix (es)  that 
are  applicable  to  all  issues. 

c.  Information  on  limitations  inherent  in  the  test 
concept,  to  include  baseline  limitations,  are  suitable  for 
inclusion.  Provisions  to  ensure  that  test  events  adhered  to 
the  operational  mode  summary  or  mission  profile  are  to  be 
included. 

d.  Provide  a  statement  regarding  test  duration  and  number 
of  operating  hours,  miles  driven,  etnd  iterations  per 
condition. 

2.4.  CONDUCT  OP  TEST.  Describe  the  conduct  of  the  test  in 
terms  of  its  tactical  context,  events,  control,  phasing, 
scheduling,  and  methodology. 
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2.4.1*  TACTICAL  CONTEXT.  Describe  the  tactical  context  and 
associated  scenario(s)  (e.g..  Middle  East,  European), 
environment,  threat,  tactics,  and  doctrine  used  in  each  test 
phase. 

2. 4. 1.1.  8CEMARI0(8). 

a.  Summarize  the  tactical  scenario (s) .  State  the  type  of 
tactical  situation  used  in  the  test.  The  scenario  description 
consists  of  the  geographic  portrayal  of  the  area,  forces,  and 
events  of  a  hypothetical  armed  conflict. 

b.  The  combat  developer  provides  standard  scenarios  as 
part  of  the  doctrinal  and  organizational  (D&O)  support 
package.  The  scenarios  provide  a  common  framework  of  selected 
situations  and  real  world  conditions  in  which  specified  test 
events  are  set  for  the  Middle  East,  Europe,  Alaska,  Korea, 
Persian  Gulf,  etc. 

2. 4. 1.2.  TEST  ENVIRONMENT. 

a.  Describe  the  environmental  conditions  such  as  the  type 
of  terrain,  climate  conditions,  and  conditions  of  the  test 
area.  Describe  the  use  of  test  site  terrain  for  each  phase, 
scenario,  mission,  or  trial. 

b.  The  tester  discusses  the  balance  between  realistic 
representation  of  the  operational  environment  and  those 
conditions  required  to  occur  during  programmed  test  events. 

He  describes  the  play  of  electronic  warfare  (EW) ,  .obscurants, 
NBC,  level  of  conflict  intensity,  mission  profiles,  and 
environmental  factors. 

2. 4. 1.3.  THREAT  FOR  TEST. 

a.  To  comply  with  AR  381-11,  a  test  must  have  used  an 
approved  threat.  Testing  must  include  an  accurate 
representation  of  the  threat  projected  to  exist  IOC+  (initial 
operational  capability  +) . 
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b.  Use  the  threat  paragraph  from  the  TEP  as  a  guide  and 
update  it  using  actual  threat  force  organization,  strength, 
and  doctrine  portrayed  in  the  test.  Explain  where  differences 
between  the  planned  and  actual  portrayed  threat  occurred. 
Attempt  in  this  paragraph  to  complete  an  accurate  description 
of  the  threat  as  portrayed  in  the  test  and  avoid  adding  a 
threat  appendix. 

c.  The  tester  describes  play  of  the  threat  systems  and 
tactics  for  each  phase.  Reference  the  threat  support  package 
and  the  system  threat  assessment  report  (STAR)  to  describe  the 
opposing  forces  (OPFOR)  tactics  and  define  the  threat  weapons 
employed  and  the  size  and  type  of  the  force. 

2. 4. 1.4.  TACTICS  AND  DOCTRINE. 

a.  This  paragraph  describes  the  friendly  tactics  and 
doctrine  to  be  played  in  each  phase.  The  phases  are  designed 
within  the  tactical  and  doctrinal  framework  of  the  D&O  support 
package. 

b.  The  tester  defines  how  the  framework  was  realized  on 
the  ground  within  the  environment  of  the  test.  He  also 
describes  the  degree  of  test  player  free  play  allowed  within 
the  framework. 

2.4. 1.5.  FRIENDLY  (PLAYER)  FORCES. 

a.  Describe  friendly  forces  used  and  types  of  units 
involved  in  the  test.  Do  not  specifically  name  a  designated 
unit.  Provide  data  on  test  player  forces  that  operated  the 
system  and  portrayed  the  supporting,  supported,  and  adjacent 
forces  in  play. 

b.  Discuss  organization  of  the  unit  and  describe  any 
significant  requirements.  Include  additional  information  such 
as  TOE  designation,  if  applicable. 
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2.4.2.  TEST  EVENTS . 

a.  This  paragraph  should  include  discussions  of  the 
organization  and  overall  layout  of  the  test  to  include 
sequence  of  phases.  Use  flow  diagrams,  time  lines,  and 
matrixes  as  appropriate  to  introduce  the  events  described. 

b.  Test  events  are  described  as  data  requiring 
occurrences  (such  as  a  helicopter  unmasking  from  a 
predetermined  location  and  attempting  to  acquire  targets 
quickly  in  a  tactically  deployed  array) .  Test  trials  may 
consist  of  many  events  which  occur  under  different  tactically 
varied  conditions  within  the  trial  (for  example,  a  free-play 
force-on-force  exercise) . 

c.  A  test  trial  is  a  continuous  tactical  exercise 
beginning  from  systematically  controlled  initial  conditions 
and  evolving  tactically  within  some  controlled  bounds  until 
meeting  some  specified  duration  or  objective.  A  test  trial  is 
the  smallest  test  planning  unit  scheduled  to  occur  under 
specific  conditions  at  a  specific  time.  Test  trials  are  made 
up  of  test  events. 

d.  Scenarios,  missions,  or  vignettes  are  built  up  from 
scheduled  trials  and  may  stretch  out  for  relatively  long 
periods  (for  example,  testing  of  an  infantry  weapon  through 
multiple  72-hour  field  exercises) . 

e.  Test  phases  are  periods  of  time  within  which  similar 
scenarios  are  conducted,  possibly  on  ranges  separated  by 
substantial  distances  (such  as  force-on-force  phase  versus 
live-fire  phase) .  Phases  usually  represent  the  largest 
breakdown  of  the  test. 

f.  Describe  the  test  phases  and  schedule  of  test  events 

as  executed  during  the  test.  An  operational  test  can  be 
composed  of  a  training  phase,  pilot  test,  several  scenario 
driven  phases,  and  specific  exercises  or  excursions.  The 
tester  defines  the  test  and,  for  each  phase  or  stratification 
of  the  test,  specifies  what  was  included  and  how  it  was 
accomplished.  Identify  locations  by  phase  when  using 
different  locations  for  specific  phases.  Include  tables  for 
event  matrixes  providing  information  on  the  major  events 
conducted  during  the  test  to  include  number,  type  and 
conditions  (light,  mission-oriented  protective  posture  (MOPP) , 
countermeasures) . _ _ _ 
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2.4.3.  CONTROL  PROCEDURES . 

a.  This  paragraph  includes  descriptions  of  the  control 
structure  and  procedures  used  to  ensure  that  required  test 
events  occurred  in  situations  that  realistically  depicted  the 
tactical  context  of  the  test  in  accordance  with  the 
operational  mode  summary  (OMS) . 

b.  Summarize  the  control  procedures.  State  if  conduct  of 
the  test  was  on  a  free-play  basis  or  if  individuals  such  as 
umpires,  control  officer,  or  operations  officer  administered 
control  of  the  test.  Summarize  the  uncontrolled  variables 
over  which  the  tester  had  no  control  such  as  player  morale, 
leadership,  or  weather,  and  the  impact  of  each  on  the  test. 

2.4.4.  SCHEDULE  OF  EVENTS.  Overall  test  schedule.  This 
information  may  appear  in  a  figure  or  table  referred  to  in 
this  paragraph. 

2.4.5.  OVERALL  METHODOLOGY. 

a.  This  paragraph  may  not  be  necessary  if  methodology  is 
fully  covered  under  each  issue  in  chapter  3  below.  When  using 
this  paragraph,  it  should  be  a  consolidation  of  methodology 
that  applies  to  all  the  issues.  This  methodology  paragraph 
should  be  the  same  as  the  overall  methodology  paragraph  in  the 
TEP  except  changing  the  verbs  to  past  tense  and  updating  the 
text  to  describe  the  test  events,  conditions,  and  methods 
actually  used  in  conducting  the  test. 

b.  State  both  the  overall  methodology  for  the  test  and 
the  methodology  common  to  more  than  one  issue.  State  issue 
specific  methodology  in  chapter  3.  As  appropriate,  update 
methodology  information  contained  in  the  TEP. 

c.  This  paragraph  should  include  descriptions  of 
execution  procedures,  control  procedures,  data  collection 
procedures,  and  any  other  information  necessary  for  an 
understanding  of  the  matrix  that  applies  to  all  issues. 

Include  information  on  assumptions,  limitations,  or  advantages 
inherent  in  the  test  concept. 

d.  Describe  the  necessity  for  testing  with  a  baseline 
system,  if  applicable. 
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2.5.  TEST  DATA  MANAGEMENT.  Provide  the  concepts  used  for 
data  management  and  the  contents  of  the  tester  data  base. 

2.5.1.  DESCRIPTION  OP  DATA  COLLECTION  METHODS.  Describe  the 
manual  and  instrumentation  methods  used  for  the  collection  of 
test  data.  Address  use  of  instrumentation,  manual  observation 
methods,  or  judgmental  or  subjective  evaluation  to  collect 
data.  Describe  provisions  for  quality  assurance  of  the  data. 

2.5.2.  DATA  BASE. 

a.  Summarize  th  i  responsibilities  of  the  Data 
Authentication  Group  (DAG) ,  if  formed.  State  what  offices 
made  up  the  DAG.  Describe  the  types  of  data  produced  during 
the  test.  Summarize  how  the  DAG  examined  the  data. 

b.  Provide  a  general  description  of  the  data  base  and 
include  a  sentence  stating  the  completion  date  of  data  base 
validation  by  the  DAG.  FDTE,  CEP,  and  CT  will  not  normally 
require  a  DAG. 

c.  When  test  data  are  extensive  enough  to  require  storage 
in  an  automated  data  base,  the  structure  and  content  of  the 
data  base  is  briefly  described  in  this  section  and  (described 
in  detail  in  appendixes  A,  B,  and  C.  The  architecture  and 
design  of  the  data  base  are  described  including  the 
relationships  among  files  and  records. 

2.6.  TRAINING  CONDUCTED.  Summarize  the  training  provided  to 
and  conducted  for  or  by  the  player  and  threat  forces  or  units. 
Identify  the  scarce  for  training  provided  to  the  units.  Also 
summarize  the  training  conducted  for  the  test  organization 
personnel,  data  collrjtors,  and  data  reducers.  Include  a 
sentence  that  states  the  date  of  the  trainer  operational  test 
readiness  statement  (OTRS)  and  any  other  pertinent  documents 
pertaining  to  training  of  personnel. 
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2.7.  INSTRUMENTATION,  TARGETS,  AND  THREAT  SIMULATORS  (ITTS) . 

a.  The  tester  should  briefly  describe  ITTS  used  in  test, 
other  than  that  used  for  data  collection.  Describe  the 
purpose  of  the  ITTS  used  and  provide  information  on  how  it 
operated  during  the  test.  Identify  decision  authority  and 
documentation  approving  any  ITTS  used  in  the  test. 

b.  Instrumentation  consists  of  electronic  devices  and 
systems  designed  to  collect,  process,  and  document  test  event 
data.  Include  discussions  of  the  following: 

(1)  Video  instrumentation. 

(2)  Existing  instrumentation  systems  used,  such  as 
range  measuring  system  (RMS) ,  TEXCOM  Automated  Field 
Instrumentation  System  (TAFIS) ,  voice  recording  system  (VRS) , 
scanning  laser  system  (SLS) ,  and  electro-optics. 

(3)  Design  and  construction  of  new  instrumentation 
devices  or  modification  of  existing  instrumentation 
devices/systems  required. 

(4)  Simulators,  stimulators,  emulators,  and  drivers 
(SSED)  for  computer  driven  systems. 

c.  When  applicable,  this  paragraph  describes  the  use  of 
existing  and  new  targets  and  threat  simulators. 

2.8.  ENVIRONMENTAL  AND  ENERGY  IMPACTS.  Include,  as  the  last 
sentence  of  this  subparagraph,  the  following  statement:  "The 
environmental  and  energy  impacts  of  this  test  (were)  (were 
not)  considered  significant.”  If  applicable,  address  the 
environmental  and  energy  impact  resulting  from  using  a  system 
during  the  test  or  of  the  test  as  a  whole.  If  the  data  are 
voluminous,  place  them  in  an  appendix  and  refer  to  the 
appendix  here. 

2.9.  TEST  OFFICER'S  OBSERVATIONS.  Include  observations  of 
knowledgeable  individuals  (e.g.,  testers,  project  officers, 
behavioral  scientists,  or  engineers)  in  this  paragraph.  Base 
the  information  contained  in  this  paragraph  on  observations  as 
well  as  test  data.  Observations  do  not  necessarily  have  to  be 
supported  by  test  data. 
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CHAPTER  3.  TEST  AND  EVALUATION  RESULTS.  (Title  this  paragraph 
"Test  and  Assessment  Results”  for  OA/EOA;  "Test  and  Analytic 
Results"  for  TER  (AE)  and  TR.) 

a.  This  chapter  is  the  most  important  chapter  in  the 
report.  This  chapter  establishes  test  results,  analyses  of 
test  data,  and  evaluative/analytic  findings.  It  should  be  a 
thorough  description  of  the  test  data  and  analytic  results 
keyed  to  the  issues,  criteria,  and  MOE/MOP  from  the  TEP. 

b.  For  full-evaluate  systems  the  evaluator  writes  this 
chapter  with  the  assistance  of  the  System  Task  Force  (STF)  to 
include  the  tester  and  test  analysts.  For  TER  (AE)  and  TR, 
the  tester  writes  this  chapter. 

3.1.  ISSUE  1.  This  is  a  restatement  of  the  first  operational 
issue  from  paragraph  1.5.  in  chapter  1. 

3.1.1.  CRITERION  1.1.  This  is  the  first  criterion  for  the 
issue  as  stated  in  the  TEP. 

3. 1.1.1.  MOE/MOP  1.1.1.  Use  the  MOE/MOP  as  stated  in  the 
TEP.  The  tester  will  support  and  provide  much  of  the 
information  to  complete  paragraphs  3. 1.1. 1.1  through 

3.1.1.1. n. 

3. 1.1. 1.1.  SPECIFIC  METHODOLOGY. 

a.  The  tester  will  prepare  the  information  to  complete 
the  methodology  paragraph. 

b.  Describe  how  data  pertaining  to  the  MOE/MOP  were 
generated,  collected,  and  reduced.  Explain  significant 
deviations  from  test  methodology  described  in  the  TEP  and 
reasons  for  not  addressing  any  required  MOE/MOP. 

c.  If  the  methodology  becomes  redundant,  place  this 
methodology  in  the  overall  methodology  paragraph  2.4.5. 

3. 1.1. 1.2.  DATA  (Level  4  Summary  Statistics). 

a.  The  tester  will  assist  the  evaluator  in  the  completion 
of  the  level  4  data  reduction  and  in  the  analysis  and 
discussion  of  the  data  results  (the  tester  writes  this 
paragraph  for  TER  (AE)  and  TR) . 
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b.  Use  tabular,  graphic  or  other  appropriate  formats  to 
show  the  descriptive  statistics  information  pertaining  to  the 
data  collected  for  the  MOE/MOP.  Show  the  source  of  the  data 
(e.g.  OT,  FDTE,  TT,  Contractor  Testing,  Model,  Simulation, 
Study,  Analysis) . 

c.  If  the  MOE/MOP  reguires  multiple  data  sources,  use  a 
separate  subparagraph  for  each  data  source.  List  data  from 
the  user  test  reported  in  the  TER/OA/EOA  first.  List  data 
from  other  data  sources  in  order  of  importance  after  the  user 
test  data  (TER  (AE)  and  TR  will  have  no  other  data) . 

(through) 


5.1.1.1. n.  ANALYSIS  AMD  DISCUSSION. 

a.  Provide  information  that  will  increase  the 
understanding  of  the  summary  statistics  data.  For  example, 
discuss  how  conditions  existing  during  the  test  may  have 
impacted  on  the  collection  or  validity  of  the  data  and  the 
impact  of  the  conditions  on  the  results. 

b.  Provide  confidence  levels  for  numerical  results.  Use 
levels  5,  6,  and  7  data  to  provide  analyses. 

3. 1.1. 2.  MOE/MOP  1.1.2. 

(through) 

3.1.1.11.  MOE/MOP  1 . 1 .  n . 

3.1.1.n<fl.  CRITERION  1.1.  FINDINGS  AND  DISCUSSION. 

a.  Provide  a  discussion  on  how  the  system  performed  in 
relation  to  the  criterion  and  to  each  of  the  MOE/MOP.  Include 
any  statistical  test  results,  analysis  or  other  information 
pertaining  to  the  performance.  State  whether  the  system  met 
or  did  not  meet  the  requirements  of  the  criterion. 

b.  Include  any  discussion  of  the  operational  significance 
of  the  finding.  Explain  how  the  results  of  the  MOE/MOP  were 
evaluated  in  determining  the  finding  for  the  criterion. 
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3.1.2.  CRITERION  1.2.  This  is  a  statement  of  the  second 
criterion  for  the  issue. 

(through) 

3.1. n.  CRITERION  l.n.  This  is  a  statement  of  the  last 
criterion  for  the  issue. 

3.1. n+l.  ISSUE  ANALYSIS  AND  DISCUSSION.  This  paragraph 
contains  any  discussion,  statistical  testing  and  analysis 
pertaining  to  the  operational  issue  considering  all  criteria, 
but  not  addressed  above  for  any  of  the  specific  criterion. 

3.1. n+2.  ISSUE  EVALUATION  AND  CONCLUSIONS.  This  paragraph 
contains  the  evaluation  for  the  issue,  to  include  evaluator 
(tester  for  TER  (AE)  and  TR)  conclusions  and  answer  to  the 
issue  question. 

3.2.  ISSUE  2.  This  is  a  restatement  of  the  second  issue  from 
paragraph  1.5. 

(through) 

3.&.  ISSUE  N.  This  is  a  restatement  of  the  last  issue  from 
paragraph  1.5. 
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CHAPTER  4.  OVERALL  CONCLUSIONS  AND  RECOMMENDATIONS  (Title 
this  paragraph  "Overall  Conclusions"  for  TER  (AE) . ) 

a.  The  following  basic  format  is  to  be  used  for  all 
documents  except  TR  of  FDTE,  CEP  and  CT.  For  TR  of  FDTE,  CEP, 
and  CT,  an  alternate  format  (see  paragraph  b.  below)  is  used. 
The  operational  evaluator  prepares  this  chapter  for  all  full 
evaluate  systems.  The  tester  assists  the  evaluator  as  a  part 
of  the  STF.  The  tester  prepares  this  chapter  for  AE  systems. 

4.1.  OPERATIONAL  EFFECTIVENESS  CONCLUSIONS  (Title 
this  paragraph  "Operational  Effectiveness 
Assessment"  for  OA/EOA.) 

(1)  This  paragraph  should  emphasize  overall 
operational  effectiveness  based  on  the  findings  in 
chapter  3  as  tempered  by  the  considered  military 
judgment  of  the  evaluator  (tester  for  AE) . 

(2)  Present  the  significant  results  and  the 
evaluation  summary  that  will  address  that  portion  of 
the  operational  issues  that  contribute  to 
operational  effectiveness. 

(3)  Include  combinations  of  issue  conclusions 
considering  mission,  need  current  and  projected 
threat,  trade-offs,  planned  improvements,  and 
potential  product  improvements. 

(4)  Address  adequacy  of  supporting  data. 

Present  a  position  on  the  future  capability  of  the 
system  to  fulfill  the  approved  requirements. 

Summarize  cons'^auences  of  fielding,  including  risks, 
burden,  and  cc  orison  to  replaced  system  as 
appropriate. 

(5)  Also  include  key  technical  evaluation 
findings  that  supplement  or  support  the  operational 
findings  or  which  present  significant  information. 

(6)  For  all  OA/EOA  use  appropriate  wording  in 
the  paragraph  to  show  that  the  evaluator  is 
assessing  progress  toward  effectiveness  requirements 
or  goals. 


Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 
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4.2  OPERATIONAL  SUITABILITY  CONCLUSIONS  (Title 
this  paragraph  "Operational  Suitability  Assessment" 
for  all  OA/EOA.) 

(1)  This  paragraph  should  emphasize  overall 
operational  suitability  based  on  the  findings  in 
chapter  3  as  tempered  by  the  considered  military 
judgment  of  the  evaluator  (tester  for  AE) . 

(2)  Present  the  significant  results  and  the 
evaluation  summary  that  will  address  that  portion  of 
the  operational  issues  that  contribute  to 
operational  suitability.  Include  combinations  of 
issue  conclusions  considering  mission,  need,  current 
and  projected  threat,  trade-offs,  planned 
improvements,  and  potential  product  improvements. 

(3)  Address  adequacy  of  supporting  data. 

Present  a  position  on  the  future  capability  of  the 
system  to  fulfill  the  approved  requirements. 

Summarize  consequences  of  fielding,  including  risks, 
burden,  and  comparison  to  replaced  system  as 
appropriate. 

(4)  Also  include  key  technical  evaluation 
findings  that  supplement  or  support  the  operational 
findings  or  which  present  significant  information. 

(5)  For  all  OA/EOA  use  appropriate  wording  in 
the  paragraph  to  show  that  the  evaluator  is 
assessing  progress  toward  suitability  requirements 
or  goals. 

4.3.  RECOMMENDATIONS.  This  paragraph  is  not  used 
in  a  TER  (AE) . 

Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 
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4.3.1.  SYSTEM  DESIGN  IMPROVEMENTS.  Base  this 
paragraph  on  shortfalls  of  the  system  and  the  future 
capability  of  the  system  to  fulfill  the  approved 
requirements.  This  paragraph  includes  the 
evaluator  recommendations  for  fixes  needed  before 
the  next  milestone,  planned  improvements,  and 
potential  product  improvements.  Include  evaluator 
suggestions  for  improvements  to  the  system  design, 
doctrine,  tactics,  organization,  and  training. The 
evaluator  should  state  his  recommendations  in  terms 
of  "what  needs  to  be  fixed”  rather  than  "how  to  fix 
it”  which  is  the  mission  of  the  MATDEV 
(CBTDEV/TNGDEV) . 

4.3.2.  FUTURE  TEST  AND  EVALUATION. 

(1)  Based  on  the  adequacy  of  this  test  and 
evaluation  and  on  system  performance,  the  evaluator 
recommends  issues  to  address  in  subsequent  studies 
or  testing.  Plan  for  follow-on  actions  to  ensure 
correction  of  identified  limitations. 

(2)  Include  summary  of  planned  product 
improvements  and  the  test  and  survey  data  to  verify 
success  of  fixes. 

b.  The  following  alternative  format  for  this  chapter  is  to 
be  used  by  the  tester  in  preparing  TR  for  FDTE,  CEP,  and  CT. 

4.1.  OVERl^L  CONCLUSIONS. 

(1)  Present  the  significant  results  and  the 
analytic  summary  that  will  address  the  issues. 

Include  combinations  of  issue  con-  sions 
considering  mission,  need,  current  end  projected 
threat,  trade-offs,  planned  improvsr.ents ,  and 
potential  improvements.  Address  adequacy  of 
supporting  data. 

(2)  Present  a  position  on  the  future  capability 
of  the  system/product  to  fulfill  the  requirements  or 
need.  Summarize  consequences  of  fielding  or 
implementation,  including  risks,  burden,  and 
comparison  to  replaced  system/product  as 

_ appropriate. _ 

Figure  6-11.  Detailed  guidance  for  development  and 

documentation  of  reporting  documents  (cont) 
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4.2.  RECOMMENDATIONS.  Base  this  paragraph  on 
shortfalls  of  the  system/product  and  the  future 
capability  of  the  system/product  to  fulfill  the 
approved  requirements/needs.  This  paragraph 
includes  recommendations  for  fixes  needed,  planned 
improvements,  and  potential  improvements.  Include 
suggestions  for  improvements  to  materiel,  doctrine, 
tactics,  organization,  and  training. 


APPENDIXES. 

a.  The  appendixes  to  the  report  provide  copies  of 
supporting  documentation,  provide  administrative  listinp,  and 
expand  report  paragraphs  into  detailed  descriptions.  The 
purpose  of  the  appendixes  is  to  provide  complete  information 
to  the  reader,  yet  avoid  interruptions  in  the  logical  flow  of 
information  in  the  main  body  of  the  document  due  to 
incorporation  of  too  much  detail. 

b.  For  this  reason,  many  of  the  appendixes  are  optional. 
Use  these  appendixes  only  if  the  information  in  the  particular 
paragraph  of  the  report  is  of  sufficient  volume  and  complexity 
to  interrupt  the  logical  flow  of  information. 

c.  The  instructions  for  each  individual  appendix  identify 
the  writer  of  that  appendix  for  each  type  of  report.  If  an 
appendix  is  not  used,  show  that  by  a  place  holder.  Do  not 
redesignate  appendixes. 

APPENDIX  A.  DATA  BASE  STRUCTURE  AND  DATA  BASE. 

a.  The  tester  always  writes  this  appendix.  This  appendix 
contains  the  detailed  description  of  the  structure  and 
contents  of  the  level  3  data  base  for  the  user  test.  It  also 
serves  to  transmit  the  level  3  data  base  to  interested  parties 
of  the  acquisition  team  and  approved  Army  and  DOD  agencies. 
This  appendix  is  not  used  for  OA/EOA  with  no  user  test. 

Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 
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b.  The  data  base  structure  and  data  dictionary  are  the 
core  of  the  data  base  description  in  the  test  and  evaluation 
report.  The  structure  of  the  data  base  is  to  be  described  in 
detail  in  the  appendix.  The  description  should  include 
identification  of  each  file  in  the  data  base,  the  relationship 
to  other  files,  and  contents.  Key  variables  which  link  files 
should  be  identified.  The  description  of  the  file  contents 
should  include,  as  a  minimum,  the  file  identification,  the 
software  type  or  format  of  the  file  (i.e.,  ASCII  (flat),  SAS, 
DBASE#,  etc.),  listing  of  each  variable  in  the  file  (to 
include  name,  description,  type,  values,  etc.),  and  total 
number  of  records  contained. 

c.  The  data  in  each  file  or  record  are  to  be  listed  and 
augmented  by  any  necessary  definitions.  Good  data  definitions 
specify  exactly  what  is  measured  when.  Examples  are: 

(1)  "Elapsed  time  to  transmit  a  call  is  to  be  measured 
and  recorded  to  the  nearest  second  by  a  data  collector  at  the 
transmitter  using  a  stopwatch.  Transmission  time  begins  when 
the  operator  . . .  <specify  operator  action>  and  ends  when  . . . 
<specify  operation>." 

(2)  "RMS  display  range  is  the  RMS  range  between  the 
aircraft  and  the  ground  target  at  the  time  when  the  1553  data 
bus  in  the  aircraft  confirms  that  the  ground  target  symbol  is 
displayed  on  the  aircraft  gunner's  scope.  This  is  available 
from  the  RMS  instrumentation  system." 

d.  Provide  data  base  structure  diagrams  which  expand  on 
data  base  descriptions.  Provide  a  crosswalk  of  data  elements 
and  the  data  collection  forms  on  which  these  data  elements 
were  collected. 

e.  The  actual  level  3  data  base  can  be  included  in  the 
appendix  if  the  data  base  is  not  too  voluminous.  If  the  data 
base  is  included  in  the  appendix,  it  should  be  no  more  than  30 
pages  of  data.  However,  because  most  of  the  data  bases  are 
not  small,  the  data  base  should  be  provided  as  a  magnetic 
medium  such  as  computer  disks  or  tapes.  The  appendix  should 
then  contain  an  inventory  of  the  number,  type,  and  contents  of 
each  tape  or  disk  with  appropriate  instructions  for  use. 

Figure  e-ITI  Detailed  guidance  for  development  and 
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f.  All  tapes  or  disks  provided  should  be  adequately 
labeled  for  reference.  If  the  data  base  is  not  provided  to 
all  agencies  on  the  distribution  list,  include  a  statement 
that  the  complete,  unabridged  data  base  is  available  upon 
request  and  the  method  and  address  (telephone  number)  for 
making  the  request. 

g.  The  contents  of  the  appendix  will  address  (include  or 
provide  on  disk/tape)  all  level  3  data  needed  to  support  the 
report.  To  assist  readers  in  locating  and  reviewing 
information  contained  in  the  appendix,  the  preparer  should 
group  the  information  by  related  material  in  separate 
paragraphs  or  sections. 

h.  Summary  (level  4  or  higher  data) . 

(1)  For  full-evaluate  systems,  normally  no  summary 
data  will  be  provided  in  the  appendix.  In  the  case  when  the 
test  agency  decides  that  a  need  exists  to  highlight  a  selected 
category  or  specific  item  of  data,  summary  tables 

no  greater  than  level  4  can  be  developed  and  included  in  the 
appendix.  Care  should  be  exercised  to  ensure  that  the  tester 
has  coordinated  the  data  summary  with  the  operational 
evaluator  prior  to  inclusion.  Such  tables  should  not 
duplicate  information  to  be  included  in  chapter  3. 

(2)  For  abbreviated  evaluate  systems  and  other  user 
tests,  data  summaries  to  supplement  or  complement  the  results 
contained  in  chapter  3  may  be  included.  Efforts  should  be 
made  to  reduce  such  summaries  to  a  minimum. 

(3)  Include  examples  of  data  forms  and  summaries  of 
responses  to  questionnaires  and  interviews  only  when  pertinent 
to  a  clear  understanding  of  the  analyses,  findings,  or 
assessments. 

APPENDIX  B.  SUPPLEMENTAL  USER  TEST  DATA. 

a.  The  tester  always  writes  this  appendix.  Include  user 
data  that  does  not  fit  into  the  data  base  (e.g.,  photos, 
videotapes,  audiotapes,  interviews).  This  appendix  is  not 
used  for  OA/EOA  with  no  user  test. 
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b.  The  tester  may  combine  this  appendix  with  appendix  A 
for  a  simple  test.  If  the  data  are  too  voluminous,  the  test 
officer  may  consider  summarizing  and  cataloging  the  data  in 
this  appendix.  If  the  data  are  not  published  as  part  of  the 
report,  include  a  statement  as  to  data  being  available  upon 
request  by  interested  agencies. 

c.  Identify  photographs  and  illustrations  by  figure 
numbers  and  captions  and  place  them  in  the  report  as  soon  as 
possible  after  first  mentioned.  Use  high-quality  photographs, 
diagrams,  and  illustrations  to  depict  test  conditions  and 
clarify  reports. 

d.  Use  arrows  to  indicate  points  of  interest  in 
photographs  and  illustrations.  Use  a  scale  or  some  other 
means  in  photographs,  when  necessary,  to  aid  the  viewer  in 
getting  the  proper  perspective  or  size  of  the  test  system  or 
item.  Obscure  trade  names  and  manufacturers'  names  on  tested 
items  in  such  photoaraphs. 

APPENDIX  C.  RAM  DATA,  COMPUTATIONS,  AND  SCORING  CONFERENCE 
MINUTES . 

a.  The  tester  writes  this  appendix  with  input  from  the 
evaluator  for  full-evaluate  system  tests  (except  for  OA/EOA 
with  no  user  test  where  it  is  written  by  the  evaluator) .  This 
appendix  is  prepared  in  total  by  the  tester  for  TER  of 
abbreviated  evaluate  systems  and  for  TR  of  FDTE,  CEP,  and  CT 
in  which  RAM  data  are  collected. 

b.  User  testing  will  assess  the  RAM  performance 
characteristics  upon  exposure  of  the  materiel  or  software  to  a 
variety  c  expected  operational  conditions.  The  scope  of  the 
RAM  tes  rg  will  vary  depending  on  the  type  of  test  or  the 
test  phase  conducted. 

c.  The  data  forms,  charts,  maintenance  records,  or  other 
RAM  data  that  are  too  voluminous  or  extensive  for  inclusion  in 
the  main  body  of  the  report  and  are  deemed  necessary  for 
understanding  of  the  RAM  area  will  be  placed  in  this  appendix. 

d.  Include  the  results  of  the  OT  scoring  conference (s)  in 

this  appendix.  Listings  of  the  score  of  each  incident  report 
may  be  included;  however,  a  summary  of  the  number  of  reports 
by  category  of  scoring  should  be  provided  prior  to  the 
listing. _ 

Figure  6-11.  Detailed  guidance  for  development  and 
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e.  Individual  test  incident  reports  (TIRs)  will  not  be 
placed  in  the  appendix  or  report. 

APPENDIX  D.  OTHER  DATA  SOURCES  (Title  this  paragraph  "Data 
Sources"  for  OA/EOA  with  no  user  test.) 

a.  The  evaluator  always  writes  this  appendix.  This 
appendix  is  not  used  for  TER  (AE)  or  TR  for  FDTE,  CEP,  and  CT. 

b.  This  appendix  contains  data  obtained  from  technical 
testing,  contractor  testing,  other  user  tests  (e.g.,  FDTE), 
modeling,  simulation,  mar)cet  surveys  and  investigations, 
studies,  and  analyses.  If  the  data  are  simple,  include  them 
here.  If  the  data  are  voluminous,  summarize  and  catalogue  the 
data  here  and  reference  where  the  data  can  be  seen  in  their 
entirety. 

c.  All  sources  of  data  to  be  used  in  an  assessment  must 
be  described  in  the  TEP.  Each  MOE/MOP  must  be  addressed  by  at 
least  one  data  source.  Any  test,  model,  simulation,  market 
survey,  market  investigation,  study,  analysis,  or 
investigation  used  as  a  primary  or  secondary  data  source  for 
any  MOE/MOP  must  be  described  in  this  appendix  to  permit 
understanding  of  the  effort.  User  tests,  contractor .tests, 
and  technical  tests  will  be  described  in  sufficient  detail  to 
provide  an  understanding  of  the  test. 

d.  Each  separate  test  and  other  effort  comprising  a 
discrete  data  source  will  have  its  own  paragraph  in  this 
appendix  and  the  paragraph  numbering  will  be  adjusted 
accordingly  (e.g.,  D.l.  FDTE,  D.2.  TTl,  D.3.  TT2,  D.4. 
Contractor  Test,  D.5.  COEA  Model,  D.6.  Other  Model,  D.7. 
Simulation,  D.8.  Market  Survey). 

D.l.  TEST  NAME  (e.g.,  FDT,  FDE,  CEP).  Use  this  paragraph  to 
describe  another  user  test  used  in  this  assessment,  the 
results  of  which  are  reported  in  a  separate  document.  Repeat 
this  paragraph  as  necessary  for  multiple  tests. ^  This 
paragraph  varies  in  scope  and  complexity  depending  on  the  size 
and  sophistication  of  the  test.  It  provides  the  basis  for 
understanding  the  value  and  context  of  the  data  as  it  applies 
to  this  assessment. 

D.1.1.  TEST  SCOPE.  Derive  the  scope  from  the  report  or  plan 
of  the  test  in  sufficient  detail  to  provide  understanding  of 
the  data  to  be  used. 

Figure  6-11.  Detailed  guidance  for  development  and 
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D.1.2.  FACTORS  AND  CONDITIONS.  Factors  and  conditions  from 
the  test  report  or  plan  are  described  in  sufficient  detail  to 
provide  a  context  for  the  data  to  be  used. 

D.2.  TEST  NAME  (e.g.,  TT,  LFT,  CONTRACTOR  TEST).  Use  this 
paragraph  to  describe  a  technical  test,  a  contractor  test,  or 
any  other  test,  demonstration,  or  review.  The  test  design 
will  be  described  in  the  test  plan  or  report  for  the  test. 
Repeat  this  paragraph  as  necessary  for  multiple  tests.  This 
paragraph  varies  in  scope  and  complexity  depending  on  the  size 
and  sophistication  of  the  test.  It  provides  the  basis  for 
understanding  the  value  and  context  of  the  data  as  it  applies 
to  this  assessment. 

D.2.1.  TEST  SCOPE.  Derive  the  scope  from  the  report  or  plan 
of  the  test  in  sufficient  detail  to  provide  understanding  of 
the  data  to  be  used. 

D.2.2.  FACTORS  AND  CONDITIONS.  Factors  and  conditions  from 
the  test  report  or  plan  are  described  in  sufficient  detail  to 
provide  a  context  for  the  data  to  be  used. 

D.3.  MODEL  OR  SIMULATION  N7D1E.  Use  this  paragraph  to 
describe  a  model  or  simulation  which  provides  data  for  the 
assessment.  Repeat  this  paragraph  as  necessary  for  multiple 
models  and  simulations.  This  paragraph  varies  in  scope  and 
complexity  depending  on  the  size  and  sophistication  of  the 
model  or  simulation.  It  provides  the  basis  for  understanding 
the  value  and  context  of  the  data  derived  from  this  source. 
Validation,  verification,  and  accreditation  sources  and  dates 
for  the  model  or  simulation  must  be  listed  in  this  paragraph. 

D.4.  MARKET  SURVEY  OR  INVES'^  ?ATION  NAME.  Use  this  paragraph 
to  describe  a  market  survey  investigation  which  provides 
data  for  the  assessment.  Describe  the  details  of  this  survey 
or  investigation  from  the  report  of  the  effort.  Repeat  this 
paragraph  as  necessary  for  multiple  surveys  and 
investigations.  This  paragraph  varies  in  scope  and  complexity 
depending  on  the  size  and  sophistication  of  the  survey  or 
investigation.  It  provides  the  basis  for  understanding  the 
value  and  context  of  the  data  derived  from  this  source. 

Figure  6-11.  Detailed  guidance  for  development  and 
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D.5.  OTHER  STUDY  OR  ANALYSIS.  Use  this  paragraph  to  describe 
a  study  or  other  analytic  effort  used  to  derive  data  for  the 
assessment.  Describe  the  details  of  this  study  or  analysis 
from  the  report  for  the  effort.  Repeat  this  paragraph  as 
necessary  for  multiple  studies  and  analyses.  This  paragraph 
varies  in  scope  and  complexity  depending  on  the  size  and 
sophistication  of  the  anticipated  study  or  analysis.  It 
provides  the  basis  for  understanding  the  value  and  context  of 
the  data  derived  from  this  source. 

APPENDIX  E.  SUPPLEMENTAL  ANALYSIS. 

a.  This  appendix  is  not  used  for  TER  (AE)  or  TR  for  FDTE, 
CEP ,  and  CT . 

b.  The  evaluator  writes  this  appendix.  Add  this  appendix 
only  if  performing  additional  analyses.  Include  evaluator 
analyses  and  excursions  performed  on  the  data,  but  not^ 
included  in  chapter  3.  Analyses  should  be  grouped  by  issue 
and  ordered  in  the  same  order  the  issues  are  addressed  in 
chapter  3 . 

APPENDIX  F.  SCENARIO. 

a.  This  appendix  is  not  used  for  OA/EOA  with  no  user 
test. 

b.  The  tester  writes  this  appendix.  For  this  appendix, 
update  the  same  scenario (s)  documented  in  the  TEP  to  reflect 
the  actual  events,  times,  locations,  and  conditions  that 
occurred  during  the  test. 

c.  Normally,  the  pilot  test  portion  of  the  scenario(s)  is 
not  shown  in  this  appendix.  However,  include  any  usable  data 
collected  during  the  conduct  of  the  pilot  test  in  the 
applicable  portion  of  the  report. 

d.  This  information  is  normally  covered  in 
paragraph  2. 4. 1.1  of  the  body  of  this  report.  The  tester 
should  add  this  appendix  if  the  information  in  the  body  of 
this  report  is  insufficient  to  fully  describe  the  scenario (s) 

I  in  detail. 

Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 
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APPENDIX  G.  SYSTEM  DESCRIPTION. 

a.  This  appendix  is  optional  if  paragraph  1.4  of  the 
report  adequately  covers  the  details. 

b.  The  tester  prepares  this  appendix  and  includes  a 
description  of  the  system  (or  combat/ training  developments 
product) .  He  should  describe  the  system  and  its  major  roles, 
missions,  and  components  or  characteristics.  A  description  of 
similarities  and  differences  between  the  system  under  test  and 
the  objective  system  under  development  is  appropriate  here. 

c.  Summarize  the  concept  for  force  structure  and 
employment.  If  a  large  amount  of  detail  is  required  to 
adequately  understand  the  system,  include  a  lengthy  system 
description  in  this  appendix. 

d.  For  OA/EOA  with  no  user  test,  the  evaluator  prepares 
this  appendix  (only  if  paragraph  1.4  of  the  OA/EOA  is 
insufficient  to  cover  the  details) .  For  TR  of  FDTE,  CEP,  and 
CT,  the  tester  includes  a  description  of  the  system/product 
under  test  in  this  appendix  (which  is  optional  if  paragraph  4 
of  the  TR  adequately  covers  the  details) . 

APPENDIX  H.  THREAT. 

a.  Paragraph  2.4. 1.3  of  the  body  of  the  TER  normally 
covers  this  information.  Add  this  appendix  only  if  the 
information  in  the  body  of  the  report  is  insufficient  to  fully 
describe  the  threat  portrayed. 

b.  The  evaluator  prepares  this  appendix  only  for  full- 
evaluate  systems.  The  tester  prepares  this  append:!  for  TER 
(AE)  and  TR  of  FDTE,  CEP,  and  CT.  They  define  the  ;roved 
threat  in  the  post-IOC  timeframe  of  the  tested  syst  and 
include  capabilities,  typical  means  of  operation,  art  known 
methods  of  defeating  the  system. 

APPENDIX  I.  TRAINING  OF  PLAYER  PERSONNEL. 

a.  This  appendix  is  not  used  for  an  OA/EOA  with  no  user 

test. 

b.  The  tester  prepares  this  appendix.  Describe  the 
training  of  player  personnel. 
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c.  For  user  testing,  the  training  of  player  personnel 
concentrates  on  providing  the  skills  necessary  to  operate  the 
equipment,  maintain  the  equipment,  and  perform  the  tactics  or 
operations  used  in  the  test.  The  results  of  the  test  will  be 
sensitive  to  the  amount  and  quality  of  player  training.  In 
fact,  for  some  tests,  the  amount  and  type  of  training  received 
by  various  individuals  or  groups  of  players  will  be  a  major 
test  variable. 

d.  For  those  tests  including  a  played  OPFOR,  the  units 
portraying  the  enemy  are  trained  to  employ  the  tactics  of  the 
threat  force.  The  employment  of  U.S.  tactics  and  procedures 
by  OPFOR  may  invalidate  or  cast  doubts  on  results  of  the  test. 

e.  For  all  tests,  document  the  actual  training  received 
as  part  of  the  final  report.  Describe  the  actual  training 
received  by  player  personnel,  as  necessary.  Where  necessary, 
include  the  subjects,  lesson  plans,  training  schedules,  time 
required,  special  training  aids  or  devices  used,  and  (when 
applicable)  special  qualifications  of  the  instructors. 

f.  Document  the  receipt  of  the  trainer's  OTRS  prior  to 
the  start  of  test.  The  trainer's  OTRS  may  be  included  in  this 
appendix. 

g.  When  the  training  information  is  extensive,  place  it 
in  this  appendix.  Paragraph  2.6  of  the  body  of  the  report 
normally  covers  this  information.  Add  this  appendix  if  the 
information  in  the  body  of  the  report  is  insufficient  to  fully 
describe  the  training  conducted. 

APPENDIX  J.  OBSERVATIONS  BY  TEST  TEAM  PERSONNEL. 

a.  This  appendix  is  not  used  for  an  OA/EOA  with  no  user 
test.  The  tester  has  the  discretion  to  add  this  appendix. 

b.  The  information  contained  in  these  assessments  is 
based  on  observations  and  does  not  necessarily  have  to  be 
supported  by  specific  test  data.  Paragraph  2.9  of  the  body  of 
the  report  may  cover  this  information. 


Figure  6-11.  Detailed  guidance  for  development  and 
documentation  of  reporting  documents  (cont) 
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APPENDIX  K.  COMMENTS  FROM  UNIT  COMMANDERS. 

a.  This  appendix  is  not  used  for  an  OA/EOA  with  no  user 
test.  The  tester  has  the  discretion  to  add  this  appendix. 

b.  Include  pertinent  comments  from  key  unit  personnel 
that  have  a  relationship  to  the  test.  The  comments  do  not 
have  to  be  supported  by  test  data  and  may  be  based  on  personal 
opinions.  Provide  appropriate  comments  by  the  commanders  of 
the  player  units  to  include  OPFOR  players. 

(through) 

APPENDIX  X.  AUTHORS  AND  SUPPORTING  PERSONNEL. 

a.  The  evaluator  prepares  this  appendix  only  for  full- 
evaluate  systems.  The  tester  prepares  this  appendix  for  TER 
(AE)  and  TR  of  FDTE,  CEP,  and  CT. 

b.  lor  historical  purposes,  they  list  all  individuals 
having  input  to  or  knowledge  of  the  writing  of  the  report  in 
this  appendix.  List  and  designate  all  members  of  the  STF. 

c.  If  the  listing  of  personnel  on  the  coordination  sheet 
is  sufficient,  add  a  placeholder  to  show  that  this  appendix  is 
"not  used." 

APPENDIX  Y.  GLOSSARY,  ACRONYMS,  AMD  ABBREVIATIONS. 

a.  The  evaluator  prepares  this  appendix  only  for  full- 
evaluate  systems.  The  tester  prepares  this  appendix  for  TER 
(AE)  and  TR  of  FDTE,  CEP,  and  CT.  The  evaluator  and  the 
tester  both  input  to  this  appendix. 

b.  This  appendix  defines  any  unusual  technical  terms  or 
frequently  used  acronyms  and  abbreviations  in  the  body  of  the 
report  or  in  other  appendixes  The  evaluator  includes  those 
terms  which  he  has  introduced  in  the  report.  The  tester 
includes  those  terms  that  he  has  introduced  which  were  not 
previously  defined  by  the  evaluator. 

c.  If  there  are  no  terms  needing  definition,  add  a 
placeholder  to  show  that  this  appendix  is  "not  used." 

Figure  6-11.  Detailed  guidance  for  development  and 
document  cion  of  reporting  documents  (cont) 
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APPENDIX  Z.  DISTRIBUTION. 

a.  The  evaluator  prepares  this  appendix  only  for  full- 
evaluate  systems.  The  tester  prepares  this  appendix  for  TER 
(AE)  and  TR  of  FDTE,  CEP,  and  CT. 

b.  The  evaluator  or  the  tester  shows  distribution  of  the 
report  by  organization  and  number  of  copies  in  this  appendix. 


Figure  6-11.  Detailed  guidance  for  development  and 
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Chapter  7 

Special  Considerations  in  Operational  Test  and  Evaluation 


Section  I 
General 


7-1.  Joint  and  multiservice  operational  test  and  evaluation 

a.  Joint  and  multiservice  T&E  is  acquisition  process  testing. 
It  is  a  type  of  JT&E  conducted  on  a  system  being  acquired  by  more 
than  one  DOD  Component.  Multiservice  T&E  planning,  conduct, 
reporting,  and  evaluation  shall  include  the  participation  and 
support  of  all  TIWG  members.  The  lead  service  will  have  overall 
responsibility  for  management  of  the  MOT&E  program  and  will 
ensure  that  Supporting  Service  requirements  are  included  in 
formulation  of  the  basic  resource  and  planning  documents.  The 
Supporting  Service  will  ensure  that  all  of  their  requirements  are 
made  known,  and  will  assist  the  lead  service  in  execution  of  the 
T&E  program.  The  lead  service  is  responsible  for  preparing  and 
coordinating  a  single  TEMP,  a  single  test  plan,  and  a  single  test 
and  evaluation  report  reflecting  the  system's  technical 
performance  and  operational  effectiveness  and  suitability  for 
each  service  component.  Coordination  of  the  TEMP  will  be  as 
depicted  in  Part  II,  this  pamphlet.  MOT&E  are  normally  initiated 
by  a  Joint  Service  Operational  Requirement  (JSOR) .  (See  AR  71- 
9.)  Testing  procedures  for  multiservice  test  and  evaluation 
follow  those  of  the  lead  service,  with  variance  as  required, 
resolved  through  mutual  agreements. 

b.  The  Memorandum  of  Agreement  (MOA)  on  MOT&E  and  JT&E, 
signed  by  the  operational  test  agency  (OTA)  commanders  provides  a 
basic  framework  and  guidelines  for  planning,  conducting, 
evaluating  and  repor  ng  T&E  involving  two  or  more  operational 
T&E  agencies.  OPTE  .  i  the  Army  proponent  for  this  MOA.  The  MOA 
is  reviewed  and  updel&d  annually  and  the  lead  is  rotated  among 
the  services. 

c.  Test  Planning.  Test  planning  for  MOT&E  will  generally  be 
accomplished  in  the  manner  prescribed  by  lead  service  directives. 
The  below  listed  general  procedures,  however,  will  be  followed: 

(1)  The  lead  service  OT&E  agency  will  begin  the  planning 
process  by  issuing  a  call  to  the  Supporting  Service  OT&E  agencies 
for  user  requirements,  critical  operational  issues  (COI) ,  and 
test  objectives. 


7-1 


DA  Pamphlet  73-1,  Part  Five,  16  October  1992 


(DRAFT) 


(2)  The  lead  service  OT&E  agency  will  consolidate  these 
user  requirements,  COIs  and  test  objectives  which  will  then  be 
approved  by  all  services  OT&E  agencies  involved  in  the  test. ^ 
Service-unique  issues  will  be  included  as  COIs  and/or  objectives. 

(3)  The  lead  service  OT&E  agency  will  accoitutnodate 
supporting  service  OT&E  requirements  and  inputs  in  the  formal 
coordination  action  of  the  TEMP.  Coordination  actions  will 
accommodate  service  unique  staffing  approval  requirements.  The 
TEMP  will  be  prepared  in  accordance  with  Volume  I  of  this 
pamphlet. 

(4)  Participating  OT&E  agency  project  officers  will  meet 
for  the  purpose  of  assigning  responsibility  for  accomplishment  of 
test  objectives  to  each  OT&E  agency.  These  assignments  will  be 
made  in  a  mutually  agreeable  manner.  Each  agency  will  then  be 
responsible  for  resource  identification  and  accomplishment  of ^ its 
assigned  test  objectives  under  the  direction  of  the  lead  service 
OT&E  agency. 

(5)  The  lead  service  OT&E  agency,  with  assistance  from  all 
participating  agencies,  will  develop  a  matrix  to  provide  a 
comparison  of  the  developer's  specifications,  user's^ 
requirements,  and  service  operational  criteria.  It  is  not  a 
source  document,  but  it  increases  management  visibility  of 
program  requirements,  increases  communications,  and  illuminates 
disconnects. 

(6)  Each  participating  agency  will  then  prepare  the  portion 
of  the  overall  test  plan(s)  for  its  assigned  objectives,  in  the 
lead  service's  test  plan(s)  format,  and  will  identify  its  data 
needs. 


(7)  The  lead  service's  OT&E  agency  will  prepare  the  MOT&E 
plans,  consolidating  the  inputs  from  all  supporting  agencies. 
After  consolidation,  the  OT&E  plans  will  be  coordinated  with  the 
supporting  services. 

d.  Test  Reporting.  The  following  test  reporting  policy  will 
apply  for  all  MOT&E  programs. 

(1)  The  lead  service  will  prepare  and  coordinate  the  report 
reflecting  the  system's  operational  effectiveness  and  suitability 
for  each  service.  It  will  synthesize  the  different  operational 
requirements  and  operational  environments  of  the  involved 
Services.  It  will  state  findings,  put  those  findings  into 
perspective,  and  present  rationale  why  there  is  or  is  not 
consensus  on  the  utility  of  the  system.  The  report  will  be 
signed  by  all  participating  services. 
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(a)  Each  participating  OT&E  agency  may  prepare  an 
independent  evaluation  report  and  a  final  test  report,  if 
reguired,  in  its  own  format  and  will  process  that  report  through 
its  normal  service  channels. 

(b)  The  participating  services*  independent  evaluation 
reports  and/or  final  test  report  will  be  appended  to  final 
reports . 


(c)  Reports,  as  required,  will  be  submitted  through  the 
lead  service’s  normal  channels  to  OSD’s  Director  of  Operational 
Test  and  Evaluation  (DOTE)  and  OSD’s  Deputy  Under  Secretary  of 
Defense  for  Research  and  Engineering,  Test  and  Evaluation 
(DDDRE(T&E))  at  least  45  days  prior  to  a  milestone  decision  or 
the  date  announced  for  the  final  decision  to  proceed  beyond  low 
rate  initial  production  (LRIP) .  An  interim  summary  OT&E  report 
shall  be  submitted  if  the  formal  end  of  test  phase  report  is  not 
available  45  days  prior  to  the  milestone  review.  A  single 
integrated  multiservice  report  will  be  submitted  90  calendar  days 
after  the  last  test  event  as  defined  in  the  MOT&E  plan.  This  may 
be  defined  as  the  final  bomb  drop,  gun  fired,  final  communication 
sequence  completed,  etc. ,  and  should  be  agreed  to  by  e 11  service 
participants. 

(2)  Interim  test  reports  will  normally  not  be  prepared. 

For  test  phases  which  extend  for  lengthy  periods,  interim  test 
reports  should  be  submitted  at  least  annually.  Test  reporting 
requirements  will  be  defined  in  the  TEMP.  When  required,  interim 
reports  will  be  prepared  in  accordance  with  lead  service’s 
directives  and  coordinated  with  all  participating  OT&E  agencies 
prior  to  release.  The  separate  OT&E  agencies  may  submit  interim 
reports  through  normal  service  channels  based  on  service— unique 
requirements,  keeping  other  participating  OTA’s  informed. 

(3)  For  those  reports  not  requiring  bmission  to  DOTE  and 
DDDRE(T&E/LFT) ) ,  a  single  multiservice  re]  is  not  required, 
but  may  be  prepared  upon  concurrence  of  ai  participants. 
Independent  evaluation  reports,  if  prepared  will  be  forwarded  to 
appropriate  commands  and  the  other  OT&E  participants  within  90 
calendar  days  after  the  last  test  event. 

(4)  The  lead  service  OT&E  agency  will  be  responsible  for 
preparing  the  appropriate  briefing (s)  which  will  be  coordinated 
with  all  participating  OT&E  agencies. 

e.  Deficiency  reporting  in  MOT&E. 

(1)  The  deficiency  reporting  system  of  the  lead  service 
normally  be  used.  All  members  of  the  multiservice  test  ttam 
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will  report  deficiencies  in  that  system.  All  information  needed 
by  the  participants  will  be  collected.  Each  deficiency  report 
will  be  coordinated  with  all  DTDs  prior  to  release.  If  the  Test 
Director  or  any  Deputy  Test  Director  disagrees  with  the  report, 
he  may  attach  an  explanation  of  his  disagreement  to  the 
deficiency  report.  The  report  will  then  be  submitted  to  the 
appropriate  developing  agency  with  that  explanation  attached. 

The  underlying  philosophy  is  that  each  participating  agency  will 
be  allowed  to  report  all  deficiencies  that  it  identifies;  the 
lead  service  will  not  suppress  those  reports.  Each  DTD  will  be 
responsible  for  submitting  deficiency  reports  into  his  own 
service's  deficiency  reporting  system  if  his  OT&E  agency  so 
requires. 

(2)  The  lead  OT&E  agency  will  ensure  a  system  is  set  up  to 
track  reported  deficiencies  and  to  provide  periodic  (monthly  is 
preferred)  status  reports  of  those  deficiencies  to  the 
participating  OT&E  agencies  and  to  the  test  team. 

(3)  Items  undergoing  test  will  not  necessarily  be  used  by 
each  of  the  services  for  identical  purposes.  As  a  result,  a 
deficiency  considered  disqualifying  (a  deficiency  deemed  to  be  of 
such  magnitude  that  the  system  will  not  meet  a  COI)  by  one 
service  is  not  necessarily  disqualifying  for  all  of  the  services. 
Deficiency  reports  of  a  disqualifying  nature  must  include  a 
statement  by  the  concerned  service  of  why  the  deficiency  has  been 
so  classified.  It  should  also  include  statements  by  the  other 
services  as  to  whether  or  not  the  deficiency  significantly 
affects  them. 

(4)  In  the  event  that  one  of  the  participating  services 
identifies  a  deficiency  that  it  considers  warrants  termination  of 
the  test,  the  circumstances  should  be  reported  immediately  to  the 
Test  Director.  All  testing  will  be  suspended  to  afford 
participating  Services  an  opportunity  to  discuss  the  deficiency. 
If  all  participants  may  determine  that  tests  can  continue  safely 
on  a  limited  basis  pending  subsequent  correction  of  the 
deficiency.  If  agreement  cannot  be  reached  concerning  the  nature 
and  magnitude  of  the  deficiency,  it  will  be  necessary  for  the 
Test  Director  to  consider  what  portions  of  the  test,  if  any,  are 
unaffected  by  the  deficiency  and  can  be  continued  safely  while 
the  deficiency  is  being  corrected.  Immediately  upon  making  such 
a  determination,  the  Test  Director  shall  provide  the  OT&E 
agencies  with  the  circumstances  concerning  the  deficiency,  the 
positions  put  forth  by  the  DTDs,  his  decision  and  reasons 
therefore. 

7-2.  OPTECs  Operational  Test  and  Evaluation  in  Support  of  U.S. 
Special  Operations  Command  (USSOCOM) 
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a.  The  nature  of  SOCOM’s  mission,  their  requirements  for 
peculiar  equipment  and  the  expediency  in  which  that  equipment  is 
needed  raise  several  issues  in  test  and  evaluation  support. 

SOCOM  and  OPTEC  must  work  together  to  conduct  operational  test 
and  evaluation  and  other  user  testing  and  in  accordance  with  the 
DOD  5000  series  directives  and  AR  73-1  and  still  meet  the 
requirements  of  SOCOM. 

b.  SOCOM' s  peculiar  acquisitions  will  be  supported  with  OPTEC 
operational  test  and  evaluation  efforts.  SOCOM  will  formally 
request  operational  test  support  on  an  as  needed  basis.  In ^ turn, 
OPTEC  will  prepare  test  plans,  conduct  tests  within  the  limits  of 
available  resources,  and  render  a  test  and  evaluation  report 
(with  up  to  level  5  data)  upon  completion.  All  related  test 
costs  will  be  assumed  by  SOCOM.  U.S.A.  John  F.  Kennedy  Special 
Warfare  Center  (USAJFKSWC)  will  serve  as  combat  and  training 
developer  representative.  Tests  requiring  U.S.  Army  support 
beyond  that  which  can  be  provided  collectively  by  OPTEC  and 
USSOCOM  will  be  resourced  in  accordance  with  the  Army's  Test 
Schedule  and  Review  Committee  (TSARC)  process.  Multiservice 
tests  are  not  included  in  this  agreement  and  will  be  conducted 
according  to  DODI  5000.2. 

c.  OPTEC  will: 

(1)  Plan  and  conduct  SOCOM  operational  and  other  user  tests 
(CT,  CEP,  EUTE,  LUTE,  lOTE,  FOTE) ,  report  results,  and  provide 
evaluations  of  each  tested  system's  operational  effectiveness  and 
suitability. 

(2)  Exercise  quality  control  over  the  planning,  execution, 
and  reporting  of  all  SOCOM  operational  test. 

(3)  Provides  operational  test  and  evaluation  guidance  and 
assistance  to  ensure  a  smooth,  responsive,  and  expedient  materie' 
acquisition  process. 

(4)  Provide  USSOCOM  a  cost  estimate  at  T-730,  365,  and  180 
days  prior  to  test  date. 

d.  SOCOM  will: 

(1)  Designate  USAJFKSWC  as  USSOCOM 's  combat  developer  for 
all  U.S.  Army  programs. 

(2)  Provide  critical  operational  issues  and  criteria  (COIC) 
for  all  Special  Operational  Forces  (SOF)  programs. 
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(3)  Request  operational  testing  support  through  HQ,  OPTEC, 
ATTN:  CSTE-MP. 

(4)  Prioritize  CEP/CT/FDTE/EUTE/LUT/IOTE/FOTE  requirements 
for  SOF  programs. 

(5)  Provide  funds  and  resources  as  required  to  support  test 
and  evaluation. 

e.  Personnel,  equipment,  facilities,  and  other  resources 
required  by  OPTEC  beyond  its  own  for  SOCOM  operational  and  other 
user  tests  will  be  identified  in  the  Outline  Test  Plan  (OTP)  and 
formally  coordinated  through  the  Test  Schedule  and  Review 
Committee  (TS ARC) /Five  Year  Test  Plan  (FYTP)  process.  This 
includes  USSOCOM  troops,  equipment,  and  facilities. 


Section  II 

OPTEC  Support  to  U.S.  Army  Training  and  Doctrine  Command  (TRADOC) 


7-3.  Operational  test  and  evaluation  and  other  user  test  in 
support  of  TRADOC 

a.  Testing  of  TRADOC  nonmateriel  products.  TRADOC,  as  a 
combat  and  training  developer  and  user  representative,  is 
dependent  upon  the  results  of  operational  T&E  conducted  by  OPTEC 
in  support  of  materiel  acquisition  and  TRADOC  Force  Development 
Evaluation.  OPTEC  will  plan  and  conduct  operational  tests  of 
materiel  systems,  specifically  including  new  and  changed  hardware 
and  software  components  and  both  system  training  devices  and  non¬ 
system  training  devices  (NSTD) . 

b.  Both  TRADOC  and  OPTEC  use  operational  T&E  results  to 
support  development  of  their  respective  positions  for  milestone 
decision  reviews  (MDR)  during  acquisition  or  change  (Materiel 
Change  (MC) )  of  materiel  systems. 

c.  TRADOC  develops  the  doctrinal,  training,  organizational, 
and  threat  aspects  of  the  "total  system"  and  provides  these  to 
OPTEC  as  test  support  packages.  Additionally,  TRADOC  is 
dependent  on  OPTEC  to  conduct  tests  and  experiments  focusing 
specifically  on  unique  TRADOC  products  of  DTLOM  in  support  of 
soldiers.  The  results  of  these  tests  and  experiments  support 
TRADOC  force  development  evaluation,  and  resultant  decisions, 
regarding  these  products. 

d.  The  Test  Schedule  and  Review  Committee  (TSARC)  serves  as 
the  mechanism  for  coordinating,  scheduling,  resourcing,  and 
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prioritizing  the  Army's  Five  Year  Test  Program  (FYTP) .  The  FYTP 
is  the  basis  for  conducting  operational  T&E  and  is  updated 
semiannually.  OPTEC  chairs  the  working  group  and  DA  General 
Officer  (GO)  TSARC  while  TRADOC  is  a  key  participant.  TRADOC 
will  participate  in  the  TSARC  working  group  meeting  to  impact 
outline  test  plans  (OTPs)  for  and  to  coordinate  priorities  of  all 
EUTE,  OT,  and  FDTE  (both  materiel  and  nonmateriel) .  The  DA 
working  group  TSARC  applies  prioritization,  methodology,  and 
builds  the  presentation  for  the  HQDA  GO  TSARC. 

e.  TRADOC  prioritizes  CEP  requirements  and  identifies 
resources  needed  to  conduct  those  CEP  and  supporting  CEP  tests 
during  the  annual  TRADOC  CEPSARC.  TRADOC  provides  OPTEC  the 
approved  CEP  priority  list  and  distributes  CEP  funds.  OPTEC 
schedules  the  CEP  test  priorities  into  the  total  test 
requirements  based  on  available  resources  and  provide  TRADOC  a 
CEP  test  schedule.  CEP  tests  are  second  in  priority  work  to  FYTP 
scheduled  tests. 

f.  T&E  mission  area  alignments  depicted  in  figure  7-1  reflect 
the  alignments  of  TRADOC  proponents.  Operational  Evaluation 
Command  (OEC)  director  tes.  Test  and  Experimentation  Command 
(TEXCOM)  directorates,  and  OPTEC  Test  and  Evaluation  Coordination 
(TECO)  offices,  to  accomplish  the  operational  T&E  mission. 

Direct  communications  among  aligned  activities  to  facilitate 
execution  of  responsibilities  and  exchange  of 
information/products  are  authorized. 

g.  TRADOC  will: 

(1)  Prioritize  EUTE,  OT&E,  and  FDTE  requirements  (those 
materiel  systems  for  which  TRADOC  is  funding  the  test  and  is  the 
combat  or  training  developer  and  major  DTLOM  products)  for 
integration  into  the  Army  FYTP  produced  by  the  DA  TSARC.  (HQ 
TRADOC,  TSJ^  ::) 

(2)  V  duct  pretest  training  of  player  personnel  in 
accordance  with  approved  Training  Test  Support  Packages  (TTSP) 
and  certify  training  readiness  for  test  participation. 

(3)  Conduct  test  planning  working  groups  for  FDTE  and,  as 
appropriate,  CEPs  and  CTs.  Provide  membership  to  TIWG  for  OT&E. 

(4)  Monitor  planning  and  development  of  materiel  developer 
System  Support  Packages  (SSP)  to  ensure  their  suitability  and 
timely  availability. 

(5)  Provide  combat  develop'^r  representation  for  all  RAM 
scoring  conferences  and  data  aucnentication  groups  (DAG) . 
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(6)  Provide  limited  subject  matter  expert  (SME)  support  for 
testing  concept  and  OTP  development  as  requested  by  OPTEC. 

(7)  Provide,  when  appropriate,  approved  standard  tactical 
scenarios  for  use  in  test  design. 

(8)  Provide  membership  to  Operational  Test  Readiness 
Reviews  (OTRR)  and  certify  test  readiness  via  Operational  Test 
Readiness  Statements  (OTRS)  for  doctrine/organization,  threat, 
and  training,  as  appropriate. 

(9)  Conduct  long  range  planning  to  identify  Concept  Based 
Requirements  System  (CBRS)  test  and  evaluation  requirements. 

(10)  Develop  the  Threat  Test  Support  Package,  associated 
appendixes,  and  OPFOR  training  if  required.  Development  and 
early  delivery  of  this  material  is  critical  when  force  on  force 
testing  is  being  conducted  or  when  OPFOR  is  employed. 

(11)  Validate  the  Threat  Test  Support  Package  and  the  test 
threat  portrayal  (CAC  Threats) . 

(12)  For  tests  and  experiments  not  involving  an  established 
PEO/PM  or  other  MATDEV  (CEP,  CT,  and  nonmateriel  oriented  FDTE) , 
initiate  request  for  Safety  Release  to  TECOM. 

(13)  Assist  OPTEC  as  appropriate  in  development  of  public 
affairs  support  plans  for  tests  scheduled  to  be  conducted  on 
TRADOC  installations. 

(14)  As  a  minimum  for  identification  of  test  requirements, 
TRADOC  will  provide  (for  FDTE,  CEP,  and  CT  [TEXCOM-test 
activity])  draft  chapter  1  of  a  TRADOC  TEP.  It  will  contain  a 
complete  set  of  OIC;  overall  test  purpose  and  scope;  background 
and/or  history;  the  system,  concept,  or  force  structure  descrip¬ 
tion  that  is  to  be  tested,  concept  of  employment,  and  any  threat 
to  be  played/portrayed. 

(15)  Identify  funds  and  resources  for  evaluation  of  TRADOC 
products . 

(16)  Monitor  testing  of  nonmateriel  products  when  deemed 
necessary. 

h.  OPTEC  will: 

(1)  Execute  all  Army  EUTE,  OT&E,  and  FDTE  prioritized  by 
the  TSARC.  Within  available  OPTEC  capacity  and  subject  to 
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availability  of  appropriate  funds,  conduct  CEP  tests  and  CT  as 
coordinated  with  TRADOC. 

(2)  Chair  the  Army  TSARC  to  prioritize  the  FYTP. 

(3)  Host  meetings  with  TRADOC  commanders/commandants  and 
other  TRADOC  representatives  to  review  test  support  provided  by 
OPTEC. 

(4)  Chair  and  provide  operational  tester  and/or  evaluator 
representation  at  RAM  scoring  conference  as  required. 

(5)  Provide  all  needed  assistance  to  TRADOC  proponent 
commanders/commandants  in  nominating  tests,  to  include  input  for 
CEP  resume  sheets  being  prepared  by  the  proponent  for  TRADOC 
CEPSARC  submission. 

(6)  In  accordance  with  AR  385-16,  obtain  a  Safety  Release 
from  Army  Materiel  Command,  Test  and  Evaluation  Command  (TECOM) , 
jyTTN:  AMSTE-ST,  for  each  operational  test  it  conducts.  Safety 
Release  must  be  available  to  test  directorate  and  TRADOC 
proponent  prior  to  all  tests  and  pretest  training  of  troops. 

(7)  Manage  the  user  test  instrumentation  program. 

(8)  For  tests /experiments  scheduled  to  be  conducted  on 
TRADOC  installations,  coordinate  public  affairs  aspects  with  the 
public  affairs  officer  of  the  host  TRADOC  installation. 

i.  Resources  and  support 

(1)  General.  TRADOC  supports  OPTEC  activities  located  on 
TRADOC  installations  with  general  housekeeping  services  (public 
affairs,  finance,  medical,  civilian  personnel,  military  person¬ 
nel,  military  justice,  base  operr  ons,  contracting,  MCA,  etc.). 
The  details  of  this  day-to-day  sv  >rt  will  be  captured  in 
appropriate  agreements  between  OP  £C  and  each  installation. 

(2)  T&E  Specific.  In  some  cases  it  may  be  necessary  to 
pool  resources  to  execute  the  Army's  operational  T&E  program. 
Specific  areas  are  discussed  below. 

(a)  Personnel,  Equipment,  Facility,  and  Other  Resources. 

(b)  Support  required  by  OPTEC  to  execute  the  Army's 
operational  T&E  program  will  be  identified  in  OTPs  and  published 
in  the  FYTP.  OPTEC  will  prepare  the  OTP  and  formally  coordinate 
with  TRADOC.  Informal  coordination  with  the  installation  tasked 
to  provide  the  support  is  encouraged. 

( 
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(c)  For  T&E  of  TRADOC  products  not  in  the  FYTP,  the 
TRADOC  proponent  will  prepare  the  cost  estimate  for  the  RS.  In 
addition,  proponent  will  coordinate  all  resources  required  to 
conduct  the  test  with  the  appropriate  MACOM  before  staffing  RS. 

( 3 )  Funds . 

(a)  OPTEC  plans,  manages,  and  executes  all  funding  to 
support  EUTE  and  OT&E.  OPTEC  funds  TRADOC  participation  in  EUTE 
and  OT&E  identified  in  OTPs  (e.g.,  CBTD^  and  TNGDEV  TDY,  mainte¬ 
nance  of  equipment,  and  test  player  training) . 

(b)  TRADOC  plans,  manages,  and  distributes  to  OPTEC  for 
execution  all  funds  to  conduct  FDTE,  CEP,  and  TRADOC  initiated 
CT.  TRADOC  prioritizes  CEP  (CEPSARC)  and  distributes  CEP  funds 
in  accordance  with  the  CEP  resume  sheet  (e.g.,  OPTEC,  TRADOC,  and 
FORSCOM  test  support) ,  procurement/ lease  maintenance  of  test  item 
(when  applicable) ,  and  TRADOC  evaluation  effort. 

(4)  Post  deployment  software  support.  TRADOC  and  OPTEC 
will  continue  to  participate,  with  the  MATDEV,  in  collocated  Post 
Deployment  software  support  (PDSS)  activities  at  Fort  Sill  and 
agree  to  pursue  similar  arrangements  there  and  elsewhere  as  long 
as  appropriate. 

7-4.  OT&E  strategy  for  Information  Mission  Area  (IMA)  and 
software  intensive  systems  (systems  developed  under  AR  70-1  with 
extensive  embedded  software  and  computer  resources).  A  new, 
flexible  strategy  has  been  developed  to  expedite  fielding  of 
software-intensive  systems.  It  is  consistent  with  the  DOD  5000 
and  DOD  7920  series  guidance,  including  the  requirement  to 
identify  LRIP  items  earlier  at  MS  II.  The  strategy  also 
implements  the  Software  T&E  Panel  (STEP)  recommendations  for  a 
unified  software  process.  This  strategy  applies  to  both  materiel 
systems  with  extensive  embedded  software,  and  IMA  systems. 
Traditional  weapon  system  OT&E  requires  the  entire  system  to 
successfully  complete  OT&E  of  production-representative  items 
before  fielding  (MS  III).  The  new  strategy  allows  fielding  of 
parts  of  software  intensive  systems,  once  successful  OT&E  of  a 
representative  sample  has  been  accomplished. 

a.  The  keystone  of  the  new  strategy  is  the  MS  Illn  approach 
shown  in  the  illustration.  (See  figure  7-2.)  The  time  line  in 
the  illustration  begins  after  MS  II.  If  a  system  has  a  hardware 
and  Commercial-Off-The-Shelf  (COTS)  software  component  (operating 
system,  communications  software,  data  base  management  system, 
query  language),  the  operational  tester  conducts  a  limited  user 
test  (LUT)  to  determine  successful  interoperability  of  the 
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hardware  and  COTS  software  and  its  interaction  with  users 
(soldiers  or  civilians)  and  the  operational  environment. 

b.  A  test  bed  must  be  configured  and  fielded  to  support  the 
LUT.  Authorization  to  purchase  and  field  the  LUT  test  bed  occurs 
at  Milestone  II  or,  in  cases  where  the  design  is  incomplete,  on 
approval  (by  HQDA  or  DoD,  depending  on  the  level  of  oversight)  of 
the  test  and  evaluation  plan. 

c.  Following  a  successful  test,  the  operational  tester  will 
redefine  the  test-bed  for  the  lOTE  of  block  1  of  the  developed 
software  to  appropriate  sites  beyond  those  required  for  the  LUT. 
The  operational  test  bed  may  increase  in  size  to  support  testing 
of  subsequent  (1  through  n)  blocks  of  developed  software. 

d.  Representative  sample. 

(1)  Each  block  of  developed  software  must  provide  added 
functionality  or  necessary  integration  capability  with  other 
systems  and  must  stand  alone,  in  the  event  that  subsequent  blocks 
are  never  fielded.  The  operational  tester  will  conduct  an  lOT 
for  each  block.  When  a  representative  sample  of  the  total 
software  functionality  to  be  developed  has  successfully  completed 
lOTE,  the  operational  evaluator  will  provide  a  fielding 
recommendation  to  the  MS  III.C  (fielding  certification)  decision 
review  body. 

(2)  To  reach  a  representative  sample,  some  number  of  blocks 
must  sufficiently  stress  the  hardware,  COTS  software,  intra¬ 
system  connectivity,  and  communications  network.  Definition  of  a 
representative  sample  will  differ  for  each  system.  Generally, 
the  representative  sample  will  be  determined  by  collating  the 
critical  mission  functions  from  the  requirements  documents  with 
the  hardware  and  COTS  software  capabilities. 

e.  Fielding  -  MS  III.C 

(1)  DOD  or  HQDA  approval  at  MS  III.C  will  allow  ti  Army  to 
authorize,  purchase  and  fielding,  to  all  users  of  100  percent  of 
the  hardware  and  COTS  software  and  all  developmental  software 
successfully  tested  to  date. 

(2)  The  OT&E  activity  will  conduct  additional  lOTEs  for 
software  blocks  developed  after  MS  III.C.  Each  block  may  be 
fielded  after  successful  lOTE.  For  each  lOTE  and  LUT,  the 
operational  evaluator  will  prepare  an  operational  assessment. 

When  the  final  block  has  completed  lOTE,  the  operational  tester 
and  evaluator  will  provide  a  Test  and  Evaluation  Report  to 
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address  the  total  system's  operational  effectiveness  and 
suitability. 

(3)  The  jagged  vertical  line  in  the  illustration  can  move 
to  the  left  or  right,  depending  on  the  definition  of  a 
representative  sample  of  the  blocks  of  software  to  be  developed. 
Many  systems  will  have  no  more  than  one  or  two  blocks;  some  may 
have  several;  regardless  of  the  design,  the  OT&E  strategy  can  be 
tailored  to  support  the  development  and  fielding  strategy. 

f .  Critical  mission  functions  (CMFs) ,  lOTE  readiness 
criteria,  and  tripwires. 

(1)  Other  features  of  the  new  strategy  include:  the 
addition  of  CMFs  to  Part  I  of  the  TEMP;  criteria  for  determining 
readiness  for  lOTE;  and  tripwires  to  determine  lOTE  requirements 
when  changes  are  made  to  the  CMFs,  hardware,  COTS  software,  or 
the  communications  network. 

(2)  CMFs  describe  the  minimum  acceptable  functionality  that 
must  be  provided  before  each  block  of  the  system  can  be  fielded. 
CMFs  are  developed  and  prioritized  by  the  user  representative  and 
are  based  on  the  user's  requirements.  CMFs  are  grouped  into  and 
enabled  by  blocks  of  developed  software.  An  example  of  a  CMF  for 
a  weapon  system  might  be  to  provide  position  location;  an  example 
for  an  IMA  system  might  be  to  process  officer  promotions. 

(3)  As  part  of  the  strategy  for  successful  fielding  of 
software— intensive  systems,  lOTE  will  not  start  without  some 
assurance  that  the  system  can  successfully  function  in  the 
operational  environment.  In  addition  to  the  standard  OT 
readiness  statements  from  the  PM,  user  representative, 
developmental  and  operational  tester  and  evaluator,  the  OT&E 
activity  will  require  the  Configuration  Control  Board  (CCB)  to 
certify  that  each  block  is  ready  for  test. 

(4)  Testing  of  changes  to  blocks  and  systems  after  fielding 
must  be  considered.  If  one  of  the  following  three  tripwires  is 
activated,  the  CCB  is  required  to  notify  the  OT&E  activity: 

(a)  Significant  impact  on  or  change  to  CMFs. 

(b)  A  computer  resource  change  that  affects  system 
operation  or  supportability. 

(c)  Changes  to  more  than  15  percent  of  the  software. 
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After  examining  the  changes  to  be  made,  the  OT&E  activity  will 
determine  whether  new  OT&E  is  required.  Otherwise,  normal  post¬ 
deployment  software  support  (PDSS)  testing  will  occur. 

7-5.  Foundations  and  history 

History  of  operational  test  and  experimentation  and  operational 
evaluation.  (See  figure  7-3,  Key  T&E  Events.) 
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Supports  Aviation  Logistics  School  as  reguired. 

For  special  operations  and  airborne  missions 
(Infantry  and  Quartermaster) ,  ABSOTD  will  be  the 
aligned  test  directorate. 

CASCOM  commander  will  consoldiate  input  pertaining  to 
selection/evalaution  of  Combat  Support  Director  at 
TEXCOM  and  selection  of  Combat  Support  Director  at 


OEC .  .  , 

Also  supports  Aviation  Logistics  School,  Chemical 
School,  Military  Police  School,  Ordnance  Center  and 
School,  Ordnance  Missile  and  Munitions  Center  and 
School,  Quartermaster  School,  Transportation  School, 
and  Soldier  Support  Center  and  associated  schools  as 
r equ ired .  _ 


Figure  7-1.  T&E  mission  area  alignments 
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Figure  7-2,OT&E  strategy  for  software-intensive  systems. 


Foundations  and  History  of  User  Test  and  Experimentation  and 
Operational  Evaluation 

a.  The  objective  of  OT&E  is  to  support  the  materiel 
acquisition  process  (MAP)  by  providing  independent  evaluations 
of  operational  effectiveness  and  suitability  based  on  all 
available  information.  OT&E  relies  on  realistic  testing  of  a 
system  when  operated  and  maintained  by  typical  users,  in  the 
same  environment  and  organization  in  which  it  is  to  be 
deployed.  OT&E  directly  contributes  to  decisions  on  whether 
to  proceed  with  developing,  procuring,  modifying,  or  deploying 
a  system.  Because  OT&E  is  an  integral  part  of  the  materiel 
development  and  decision  making  process,  it  must  be  responsive 
to  program  objectives,  milestones  (MS),  and  acquisition 
strategies  (AS) .  The  structured  approach  described  in  this 
pamphlet  applies  to  all  OT&E. 

b.  Current  OT&E  practices  have  evolved  over  many  years. 

The  goal  of  these  continuing  efforts  has  been  to  provide 
improved  comprehensive  system  evaluations  in  support  of  the 
MAP.  In  May  1970,  the  Department  of  Defense  (DOD)  recognized 
the  need  for  an  independent  estimate  of  the  operational 
effectiveness  and  suitability  of  materiel  systems.  DOD 
recommended  that  each  DOD  component  establish  an  agency 
responsible  for  OT&E  separate  from  the  developing,  procuring, 
and  using  commands.  After  the  1970  Blue  Ribbon  Defense  Panel 
report  to  the  President  and  the  Secretary  of  Defense  (SECDEF) 
the  Army  established  the  Operational  Test  and  Evaluation 
Agency  (OTEA) .  The  following  10  major  documents  and  events 
have  shaped  the  OTEA  mission. 

(1)  Blue  Ribbon  Defense  Panel.  In  July  1969,  a  Blue 
Ribbon  Defense  Panel  was  appointed  by  the  President  and  SECDEF 
to  study  the  organization,  structure,  and  operation  of  DOD. 

As  a  result  of  panel  recommendations,  OTEA  was  established  in 
1972  and  assigned  responsibility  for  operational  testing  (OT) 
and  providing  independent  evaluations  of  systems  based  on 
those  tests.  The  panel  found  that  OT&E  should: 

(a)  Consider  the  interface  with  other  systems  and 
equipment,  tactics  and  techniques,  organizational 
arrangements,  and  the  human  skills  and  capabilities  of  the 
eventual  users. 

(b)  Contribute  more  to  materiel  acquisition  decisions, 

be  conducted  in  a  realistic  operational  environment  with 
typical  user  troops,  and  be  independent  of  the  developing, 
procuring  and  using  commands. _ — 

- -  Figure  7-3.  Key  T&E  events 


(2)  Department  of  Defense  Directive  (DODD)  5000.3 
replaced  by  Department  of  Defense  Instruction  (DODI)  5000.2. 
Each  revision  of  DODD  5000.3  increased  emphasis  on  thorough 
system  OT&E  to  support  decision  makers.  DODD  5000.3  (1973) 
directed  each  DOD  component  to  establish  a  major,  separate 
field  agency  to  conduct  OT&E.  It  directed  each  service's 
operational  test  activity  (OTA)  to  report  the  operational 
effectiveness  and  suitability  of  developing  systems  directly 
to  its  military  service  chief.  The  1978  revision  emphasized 
establishing  critical  issues,  test  criteria,  and  measures  of 
effectiveness  (MOEs)  early  in  the  acquisition  cycle. 
Operational  Test  Activities  (OTAs)  objectives  were  expanded  to 
include  survivability,  vulnerability,  transportability, 

human  factors,  compatibility,  interoperability, 
reliability,  availability,  maintainability,  logistics 
supportability,  and  training  requirements.  OT&E  agencies  were 
charged  to  participate  in  the  initial  planning  for 
developmental  testing  and  evaluation  (DT&E) ,  including 
participating  in  the  early  stages  of  software  planning  and 
development.  The  1979  directive  emphasized  planning 
documentation  for  test  and  evaluation  (T&E)  and  provided 
guidelines  for  preparing  the  Test  and  Evaluation  Master  Plan 
(TEMP) .  The  1986  directive  emphasizes  the  use  of  analysis, 
modeling,  and  simulation  to  complement  developmental  testing 
(DT)  and  OT.  It  also  directed  the  developer  and  the  agencies 
to  establish  databases  for  sharing  information  and  data  during 
the  acquisition  process.  It  mandates  the  defining  of  critical 
T&E  issues,  objectives,  methodologies,  and  evaluation  criteria 
while  the  acquisition  program  is  being  established. 

(3)  Army  Materiel  Acquisition  Review  Committee  (AMARC) 
Report.  AMARC  was  formed  in  December  1973  to  review  and 
analyze  the  MAP.  As  a  result,  AMARC  recommended  that  the 
Department  of  the  Army  (DA) : 

(a)  Emphasize  the  technical  orientation  of  DT  and  the 
operational  orientation  of  OT. 

(b)  Stress  independent  test  design  »  evaluation 
rather  than  separate  DT  and  OT  as  a  solution  to  the  problem. 

(c)  Have  OTEA  report  directly  to  the  Chief  of  Staff, 
Army  (CSA) . 

(d)  Expand  the  OTEA  mission  to  include  oversight  of 
significant  nonmajor  systems. 

(4)  Letter  of  the  Vice  Chief  of  Staff,  Army  (VCSA) .  This 
August  1983  letter  directed  OTEA  to  track  the  correction  of 
deficiencies  for  major  systems  and  certain  acquisition 
programs  throughout  the  MAP  and  to  summarize  periodically  the 
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(5)  Government  Accounting  Office  (GAO)  Report.  The  GAO 
Report,  "The  Army  Needs  More  Comprehensive  Evaluations  To  Make 
Effective  Use  of  Its  Weapon  Systems  Testing,"  February  1984, 
recommended  that  one  principal  evaluation  agency  be 

Army's  overall  evaluator.  This  agency  would  integrate  both 
DTiE  and  OT&E  results  to  provide  a  balanced,  coherent  view  or 
a  system's  development  and  operational  readiness.  Based  on 
the  GAO  report  and  a  2  November  1983  meeting  between  VCSA  and 
the  Deputy  Chiefs  of  Staff  for  Research,  Development,  and 
Acquisition  (DCSRDA)  and  Operations  and  Plans  (DCSOPS),  the 
CSA  designated  OTEA  as  the  Army's  comprehensive  evaluator  in 
March  1984  and  tasked  OTEA  to  begin  Continuous  Comprehensive 
Evaluation  (C2E)  on  five  pilot  programs  with  assistance  from 
the  Training  and  Doctrine  Command  (TRADOC)  and  the  Army 
Materiel  Command  (AMC) . 

(6)  Test  and  Evaluation  Functional  Area  Analysis.  On  13 
May  1985,  VCSA  chaired  the  T&E  Functional  Area  Assessment 
(FAA) ,  ended  the  pilot  C2E  program,  and  directed  that  C2E  be 
started  on  other  systems  as  appropriate. 

(7)  AR  71-3,  User  Testing,  1  March  1986  and  draft  DA 
Pamphlet  71-  3,  chapter  12  published  December  1988. 

(8)  Army  Test  and  Evaluation  Implementation  Plan  Review. 
The  Under  Secretary  of  the  Army  (USA)  and  VCSA,  review  of  2 
March  1988,  designated  the  Test  and  Experimentation  Command 
(TEXCOM)  as  responsible  for  operational  testing  of  all 
categories  of  systems.  OTEA  (now  OPTEC)  would  selectively 
test  designated  systems  and  be  responsible  for  the  operational 
evaluation  of  all  Army  systems  unless  the  operational  test 
results  were  sufficient  to  support  the  full-rate  production 
decision.  Test  reports  on  systems  for  which  OTEA  was  not  the 
evaluator  would  be  expanded  to  include  evaluation  information; 
those  expanded  reports  would  contain  an  abbreviated  evaluation 

by  OTEA. 

(9)  Defense  Management  Review  Decision  Number  936  dated 
20  November  1989  directed  the  Army  to  consolidate  the  US  Army 
Operational  Test  and  Evaluation  Agency  (OTEA)  in  Alexandria, 
Virginia  with  the  TRADOC  TEXCOM  located  at  Fort  Hood,  Texas 
into  the  U.S.  Army  Operational  Test  and  Evaluation  Coiomand 
(OPTEC) .  The  mission  of  the  new  command  was  to; 

(a)  Support  the  Army  acquisition  process  by  managing 
the  Army's  continuous  evaluation  and  user  test  programs. 

(b)  Manage  and  execute  the  Army  user  test  process  by 
planning,  conducting  and  reporting  on  T&E  programs. 


(c)  Execute  user  tests  to  support  doctrine,  training 
and  force  design  of  the  Concept  Based  Requirements  System 
(CBRS) ,  (See  TRADOC  Regulation  11-15) ,  in  accordance  with 
TRADOC  requirements  and  TSARC  established  priorities. 

(10)  DOD  Directive  5000.1,  Defense  Acquisition,  23 
February  1991,  DOD  Instruction  5000.2,  Defense  Acquisition 
Management  Policies  and  Procedures,  23  February  1991,  and  DOD 
Manual  5000. 2M,  Defense  Acquisition  Management  Documentation 
and  Reports,  23  February  1991.  The  three  DOD  documents  are 
directive  in  nature  and  apply  to  the  Military  Departments  (DOD 
Components)  for  the  management  of  major  and  nonmajor  defense 
acquisition  programs.  The  Army  further  delineates  specific 
Army  test  and  evaluation  policy  in  AR  73-1,  Test  and 
Evaluation  Policy  dated  December  1991. 
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Chapter  1 
Overview 


Section  I 
Introduction 


1-1 .  General 

Through  a  series  of  amendments  to  Title  10,  United  States  Code, 
Congress  has  mandated  that  major  weapon  system  and  munition 
programs  undergo  a  realistic  Live  Fire  Test  and  Evaluation  (LFT&E) 
program.  This  volume  attempts  to  achieve  the  following: 

a.  Present  the  basis  for  determining  whether  a  LFT&E  program 
is  required  for  a  given  system. 

b.  Describe  the  key  steps  in  developing  an  adequate  and 
acceptable  LFT&E  strategy  including  the  role  of  modeling  and 
testing  in  the  LFT&E  process. 

c.  Provide  guidance  on  the  planning,  execution,  reporting,  and 
review  and  approval  processes  for  LFT&E  programs. 

d.  Outline  the  roles  of  key  LFT&E  activities. 
i-2.  Basic  Elements 

Figure  1-1  illustrates  the  basic  elements  of  the  overall  LFT&E 
process  from  initial  strategy  definition  to  the  writing  of  the 
final  test  and  evaluation  report ( s) .  While  the  details  of  each 
element  of  this  overall  process  must  be  decided  on  a  case-by-case 
basis,  this  volume  will  attempt  to  provide  the  foundation  required 
to  develop  a  credible  LFT&E  program.  It  draws  upon  those  general 
approaches  and  lessons  learned  from  initial  LFTs  which ^ have ^ proven 
successful  and  which  should  prove  beneficial  to  those  individuals 
involved  in  future  LFT&E  programs. 


Section  II 
Why  LFT&E? 


1-3 .  General 

As  stated  previously,  LFT&E  is  necessary  because  it  is  the  law; 
but,  more  importantly,  it  is  necessary  because  it  is  cost  effective 
and  smart  testing  (i.e.,  it  simply  makes  sense).  A  realistic  LFT&E 
program  represents  the  best  alternative  to  "actual"  combat  in 
assessing  our  systems  performance  and  is  more  cost  effective  than 
combat.  However,  with  the  lack  of  actual  combat  data  must  come  a 
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disciplined  and  realistic  approach  to  the  assessment  of  the 
vulnerability  and  lethality  of  our  weapon  systems.  The  LFT&E 
program  provides  the  needed  means  for  assessing  the  synergistic 
effects  of  system  component  integration  and  of 

selected  damage  mechanisms.  A  well-planned  and  well-structured 
LFT&E  program  reduces  the  potential  for  "surprises”  prior  to  that 
system's  arrival  on  the  battlefield. 

(Insert  Figure  1-1) 

1-4.  LFT&E  An  Essential  Element 

Furthermore,  an  active,  well-planned,  well-managed,  and  well- 
executed  LFT&E  program  is  essential  to  understanding  system 
vulnerability/ lethality  and  will  be  an  essential  element  of  the 
information  supporting  decisions  regarding  the  acquisition  of 
materiel  as  well  as  the  development  of  doctrine  and  plans  for  its 
proper  operational  employment.  When  properly  structured  and 
scheduled,  the  LFT&E  program  will  enable  design  changes  resulting 
from  that  testing  and  analysis  to  be  incorporated  into  the  system 
at  the  earliest  possible  date  and  reduce  the  need  for  expensive 
retrofit  programs. 


Section  III 
Objective  of  LFT&E 


1-5.  Objective 

The  objective  of  LFT&E  is  to  support  a  timely  and  thorough 
assessment  of  the  vulnerability/ lethality  of  a  system  as  it 
progresses  through  its  development  and  subsequent  production 
phases.  The  LFT&E  program  should  demonstrate  the  ability  of  the 
weapon  system  or  munition  to  provide  battle  resilient  survivability 
or  lethality  and  provide  insights  into  the  principal  damage 
mechanisms  and  failure  modes  occurring  as  a  result  of  the 
munition/ target  interaction  and  into  techniques  for  reducing 
personnel  casualties  or  enhancing  system  survivabi-  ty/lethality. 
These  insights  will  mature  during  the  course  of  tl  i...FT&E  program. 
Data  will  emerge  which  will  identify  specific  failure  and  damage 
mechanisms.  With  this  knowledge,  cost  effectiveness  trade-offs  can 
be  conducted  to  predict  the  optimal  "mix"  of  vulnerability 
reduction/ lethality  enhancement  measures. 

1-6.  Primary  Emphasis 

The  primary  emphasis  of  LFT&E  is  on  realistic  testing  as  a  source 
of  personnel  casualty,  vulnerability,  and  lethality  information  to 
ensure  potential  design  flaws  are  identified  and  corrected  prior  to 
full-rate  production.  The  LFT&E  program  should  provide  an 
assessment  of  a  system's  vulnerability/ lethality  performance 
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relative  to  the  expected  spectrum  of  battlefield  threats;  it  is  not 
constrained  to  addressing  specific  design  performance  goals  or 
threats.  (However,  LFT&E,  by  itself,  is  not  a  basis  for  the 
decision  to  transition  to  full-rate  production;  many  other  factors 
must  be  considered  in  arriving  at  this  decision.)  Additionally, 
LFT&E  will  provide  insights  into  how  to  enhance  the  survivability 
and/or  lethality  of  similar  or  future  systems  and  provide  a 
mechanism  for  gaining  insights  into  the  adequacy  of 
vulnerability/ lethality  assessment  techniques  and  supporting  data 
bases. 


Section  IV 
Background 


1-7.  Genesis 

The  genesis  of  LFT  began  in  the  early  1980s  and  was  the  outgrowth 
of  perceived  needs  by  two  separate  groups.  First,  the 
vulnerability/ lethality  assessment  community  was  concerned  that  the 
technological  viability  of  their  assessment  techniques  was  becoming 
increasingly  tenuous  because  they  were  relying  more  and  more  on 
questionable  extrapolation  of  existing  data  bases  (rapid  advances 
in  technology  over  the  past  two  decades  had  simply  made  many  of 
these  data  bases  outdated  and  inapplicable) .  Due  to  the  increasing 
complexity  of  foreign  and  domestic  weapon  systems  and  of  the 
munition/target  interaction,  assessment  techniques  demand  a  strong 
tie  to  empirical  data  bases  including  those  based  on  firings 
against  full-up  targets.  Staff  personnel  within  Congress,  Office 
of  the  Secretary  of  Defense  (OSD) ,  and  Headquarters,  Department  of 
the  Army  (HQDA)  were  concerned  that  testing  programs  were  ignoring 
the  realities  of  war  and  were  not  providing  a  realistic  and 
rigorous  assessment  of  the  likely  performance  of  these  systems  in 
combat.  They  felt  that  program  decisions  were  too  dependent  on 
modeling  and  component  testirg  and  that  full-up  LFT  was  needed  to 
judge  how  well  these  system?  ' nd  the  crew  who  operated  them — would 
survive  on  the  modern  battle,  .eld. 

1-8.  Joint  Live  Fire  Program 

The  need  for  full— up  testing  led  to  the  establishment  of  the  Joint 
Live  Fire  (JLF)  Program  in  March  1984;  the  JLF  Program  was  and 
continues  to  be  sponsored  by  OSD  as  a  joint  test  initiative.  The 
JLF  Program  is  chartered  to  assess  the  vulnerabilities  and 
lethalities  of  fielded  conventional  U.S.  ground  systems  and 
aircraft.  Army  systems  initially  included  in  the  JLF  Program  were 
the  Bradley  Fighting  Vehicle  System,  the  Abrams  Tank,  and  the  M113 
Family  of  Vehicles.  Because  of  difference  in  the  philosophic 
approach  to  LFT  between  the  Army  and  OSD  (the  building-block 
approach  versus  large  scale  full-up  testing)  and  ti^e  Army’s  desire 
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to  accelerate  the  testing  of  these  systems,  the  Army  subsequently 
requested  and  received  permission  from  OSD  to  withdraw  the  Bradley, 
Abrams,  and  M113  systems  from  the  JLF  Program.  The  Army  agreed  to 
fund  the  cost  of  the  LFT  programs  for  these  systems  and  to  provide 
open  access  by  OSD  to  test  planning,  test  conduct,  and  test 
results.  This  series  of  LFTs  was  known  as  Army  LFT  and  was 
completed  in  1988 . 

1-9 .  Controversy 

Controversy  over  the  basic  approach  to  LFT  and  the  Army|s  conduct 
of  the  Bradley  Live  Fire  Test  (the  perception  of  test  biasing) 
highlighted  the  need  for  LFT  and  led  Congress  to  mandate  such 
testing  for  major  weapon  system  and  munition  programs  through  a 
series  of  amendments  to  Title  10,  United  States  Code  in  the 
FY86  through  FY91  Department  of  Defense  (DoD)  Authorization  Acts. 
Table  1-1  presents  a  comparison  of  the  primary  features  and 
differences  among  the  JLF,  the  Army  Live  Fire,  and  the 
Congress ionally  legislated  LFT&E  programs.  The  remainder  of  this 
part  is  devoted  to  a  discussion  of  the  requirements  and  strategies 
applicable  to  Congressionally  legislated  LFT&E  programs. 

Section  V 
LFT&E  Legislation 


1-10.  Title  10 

The  FY86  and  FY87  DoD  Authorization  Acts  amended  Chapter  139  of 
Title  10,  United  States  Code,  to  require  LFT&E  prior  to  proceeding 
beyond  low-rate  initial  production  (LRIP) .  Specifically,  the  FY86 
legislation  requires  side-by-side  vulnerability  LFT&E  if  a  wheeled 
or  tracked  armored  vehicle  is  to  replace  an  existing  vehicle;  the 
FY87  legislation  requires  LFT&E  for  all  covered  systems  and  major 
munition  and  missile  programs.  The  FY88-89  DoD  Authorization  Act 
amended  Title  10  to  include  a  LFT&E  requirement  for  product 
improvements  to  major  systems 

(i.e.,  materiel  changes  (MC) ) ;  the  FY90-91  Act  requires  DoD  to 
report  results  of  LFT  before  a  system  enters  full-rate  production 
and  also  acknowledges  that  procurement  funds  can  be  used  to  support 
LFT&E  programs  (such  funding  shall  not  exceed  one-third  of  one 
percent  of  the  total  program  cost) . 

1-11.  Current  Legistlation 

To  summarize,  the  current  legislation  requires  that  the  Secretary 
of  Defense  provide  that: 

a.  A  covered  system  not  proceed  beyond  LRIP  until  realistic 
survivability  testing  is  completed. 
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Table  1-1.  Comparison  of  Joint  Live  Fire,  Army  ^jive  Fire, 
and  Congress ionally  Legislated  LFT&E  Programs 


Joint  Live  Fire 


Chartered  FY84 
Multiservice 
OSD  Funded 
Fielded  Systems 

Vulnerability/ 
Lethality 
Armor / Ant i -Armor 
Aircraft 
Test  Event 
Oriented 
OSD  Oversight 


Army  Live  Fire 


Legistlated/Chartered 
Army  Only 
Army  Funded 
Bradley , Abrams , H113 
Family 

Vulnerability 

Armor 

Test  Event  Oriented 
OSD  Oversight 


Conaressionallv 
Leaistlated  Live  Fire 

Legistlated  FY86-FY89 
Individual/Multiservice 
Service  Funded 
Developmental  Systems 
/PIPS 

Vulnerabi 1 ity / 

Lethality 

Air,  Land, Sea  Systems 
Milestone  Oriented 
OSD  Oversight 
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b.  A  major  munition  or  missile  program  not  proceed  beyond  LRIP 
until  realistic  lethality  testing  is  completed. 

c.  A  covered  PIP  not  proceed  beyond  LRIP  until  realistic 
survivability /lethality  testing  is  completed. 

The  legislation  states  that  the  costs  of  all  survivability/ 
lethality  testing  shall  be  paid  from  funds  available  for  the  system 
being  tested.  The  legislation  also  provides  that  the  Secretary  of 
Defense  may  waive  the  requirement  for  survivability/ lethality 
testing  in  the  time  of  war  or  if  the  Secretary  certifies  to 
Congress,  before  the  system  enters  engineering  and  manufacturing 
development  (EMD)  (formerly  full-scale  development  (FSD) )  that  LFT 
of  that  system  would  be  unreasonably  expensive  or  impractical. 

NOTE:  Per  Departmertt  of  Defense  Instruction  (DoDtl  5000.2,  23  February  1991,  all  acquisition 
programs,  excluding  highly  classified  programs,  shall  be  placed  into  one  of  four  categories.  Acquisition 
Category  (A  CA  V  I,  A  CAT  II,  A  CAT  III.  or  ACAT IV.  ACAT I  and  A  CAT  II  programs  are  major  defense 
acquisition  programs  and  major  programs,  respectively,  and.  if  they  are  covered  systems  or  a 
munition/missile  system,  will  have  a  LFT&E  requirement.  ACAT  III  and  ACAT  IV  munition/missile 
programs  may  have  a  LFT&E  requirement  if  they  the  meet  the  1 ,000,000  round  production 
requirement. 

1-12.  LFT&E  Requirements 

Figure  1-2  provides  a  flow  chart  to  assist  in  determining  a  systems 
LFT&E  requirement.  This  flow  chart  addresses  both  new  systems  and 
materiel  changes  (PIPs)  to  existing  systems.  Specific  situations 
(e.g.,  the  LFT&E  requirements  for  "add-ons"  to  existing  systems 
which  have  undergone  LFT&E)  must  be  addressed  on  a  case-by-case 
basis.  Basically,  if  a  system  meets  the  LFT&E  dollar  or  quantity 
criteria  or  if  a  PIP  provides  a  significant  V/L  effect,  the  system 
has  a  LFT&E  requirement.  The  degree  of  LFT&E  needs  to  be  addressed 
in  a  comprehensive  LFT&E  strategy,  incorporated  into  the 
appropriate  documentation,  and  provided  the  Army  leadership  for 
guidance  and  approval.  Per  DoDI  5000.2,  a  systems  proposed 
acquisition  strategy  developed  during  Acquisition  Phase  0  (Concept 
Exploration  and  Definition)  "must  include  provisions  for  conducting 
live  fire  testing  on  covered  systems,  major  munition  programs  and 
missile  programs  (and  covered  product  improvements  programs 
thereto)";  Army  policy  requires  a  system’s  LFT&E  requirement  be 
identified  to  TEMA  and  the  initial  strategy  and  resource 
requirements  be  included  in  the  Milestone  I  Test  and  Evaluation 
Master  Plan  (TEMP) . 

(Insert  Figure  1-2) 

Section  VI 
Definitions 

1-13.  Definitions. 
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NOTE:  Dollar  quantities  are  FY80  constant  dollars. 

Figure  i^t,  LFT&E  Reqpiirenent  Flow  Chart. 
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The  following  definitions  of  commonly  used  terms  are  presented  here 
in  order  to  provide  a  better  understanding  of  the  LFT&E  process: 

a.  Survivability.  The  capability  of  a  system  to  avoid  or 
withstand  a  man-made  hostile  environment  without  suffering  an 
abortive  impairment  of  its  ability  to  accomplish  its  designated 
mission. 


b.  Vulnerability.  The  characteristics  of  a  system  that  cause 
it  to  suffer  a  definite  degradation  (loss  or  reduction  of 
capability  to  perform  its  designated  mission(s))  as  a  result  of 
having  been  subjected  to  a  certain  level  of  effects  in  a  man-made 
hostile  environment. 

c.  Lethality.  The  ability  of  a  munition  to  cause  damage  that 
will  cause  the  loss  of  or  a  degradation  in  the  ability  of  a  target 
system  to  complete  its  designated  mission (s). 

d.  Covered  System.  Any  vehicle,  weapon  platform,  or 
conventional  weapon  system  that  includes  features  designed  to 
provide  some  degree  of  protection  to  users  in  combat  and  is  a  major 
system  (see  paragraph  5.1.6.g  below  for  the  definition  of  major 
system) . 

e.  Major  Munitions  Program.  A  conventional  munitions  program 
that  is  a  major  system  within  the  definition  given  in  paragraph  1- 
13. g  or  for  which  more  than  1,000,000  rounds  are  planned  to  be 
acquired. 

f .  Covered  Product  Improvement  Program  (PIP) .  A  covered 
system  and/or  major  munition  or  missile  program  for  which  a  planned 
modification  or  upgrade  is  likely  to  produce  a  significant  effect 
on  the  vulnerability  and/or  lethality  of  that  system/munition  or 
missile. 


g.  Major  System.  As  specified  in  Title  10,  United  States 
Code,  Section  2302(5),  a  major  system  means  a  combination  of 
elements  that  will  function  together  to  produce  the  capabilities 
required  to  fulfill  a  mission  need.  The  elements  may  include 
hardware,  equipment,  software  or  any  combination  thereof,  but 
excludes  construction  or  other  improvements  to  real  property .  A 
system  shall  be  considered  a  major  system  if: 

(1)  The  DoD  is  responsible  for  the  system  and  the  total 
expenditures  for  research,  development,  test  and  evaluation  for  the 
system  are  estimated  to  be  more  than  75  million  dollars  (based  on 
fiscal  year  1980  constant  dollars) ,  or  the  eventual  total 
expenditure  for  procurement  of  more  than  300  million  dollars  (based 
on  fiscal  year  1980  constant  dollars) . 
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(2)  A  civilian  agency  is  responsible  for  the  system  and  the  total 
expenditures  for  the  system  are  estimated  to  exceed  750,000  dollars 
(based  on  fiscal  year  1980  constant  dollars)  or  the  dollar 
threshold  for  a  "major  system"  established  by  the  agency  pursuant 
to  Office  of  Management  and  Budget  (0MB)  Circular  A-109,  entitled 
"Major  Systems  Acquisitions,"  whichever  is  greater. 

(3)  The  system  is  designated  a  "major  system"  by  the 
Secretary  of  the  Army. 

NOTE:  Per  DoDI  5000.2,  fiscal  year  1990  cortstant  dollars  are  1 15  million  dollars  for  research, 
development,  test  and  evaluation,  and  540  million  dollars  for  procurement. 

h.  Realistic  Testing.  For  vulnerability  testing:  the  firing 
of  munitions,  likely  to  be  encountered  in  combat,  at  the  weapon 
system  configured  for  combat.  For  lethality  testing:  the  firing 
of  the  munition  or  missile  concerned  at  appropriate  targets 
configured  for  combat. 

i.  Live  Fire  Test.  A  test  event  within  an  overall  LFT&E 
strategy  which  involves  the  firing  of  actual  munitions  at  target 
components,  target  sub-systems,  target  sub-assemblies,  and/or 
sub-scale  or  full-scale  targets  to  examine  personnel  casualty, 
vulnerability,  and/or  lethality  issues. 

j.  Full-up  Testing.  Firings  against  full-scale  targets 
containing  all  of  the  dangerous  materials  (e.g.,  ammunition,  fuel, 
hydraulic  fluids,  etc.),  system  parts  (e.g.,  electrical  lines  with 
operating  voltages  and  currents  applied,  hydraulic  lines  containing 
appropriate  fluids  at  operating  pressures,  etc.),  and  stowage  items 
normally  found  on  that  target  when  operating  in  combat.  Full-up 
testing  includes  firings  against  full-up  components,  full-up 
sub-systems,  full-up  sub-assemblies,  or  full-up  systems.  The  term 
"full-up  testing"  is  synonymous  with  "realistic  survivability 
testing"  or  "realistic  lethality  testi^  a”  as  defined  in  the 
legislation  covering  LFT. 

k.  Model/Modeling.  A  vulnerability /lethality  assessment  tool 
used  to  predict  one  or  more  aspects  of  a  given  munition/ target 
interaction.  A  model  may  be  anything  from  a  sophisticated  computer 
code  (employing  many  individual  algorithms  to  assess  total  system 
vulnerability/ lethality)  to  a  simple  mathematical  expression  or 
empirical  relationship  used  to  predict  a  single  element  of  a 
munition/target  interaction  (e.g.,  the  penetration  performance  of  a 
given  munition) . 

l.  Pre-Shot  Prediction.  An  S  priori  prediction  of  the 
expected  outcome (s)  of  a  live  fire  shot.  The  prediction  might,  in 
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special  circumstances,  be  a  quantified  value  of  the  probability  of 
kill  given  a  hit  and/or  the  expected  number  of  casualties.  Most 
often,  the  pre-shot  prediction  will  be  in  the  form  of  quantitative 
or  qualitative  expectations  of  the  ability  of  the  attacking 
munition  to  defeat  the  armor  or  other  protective  design  features  of 
the  target  and  inflict  damage  to  components  or  personnel;  or 
conversely,  the  ability  of  the  target  to  defeat  or  mitigate  the 
effects  of  the  attacking  munition.  These  predictions  can  be  either 
absolute  expectations  of  performance  or  comparative  expectations  of 
the  relative  performance  of  two  or  more  munitions  or  targets.  The 
pre-shot  predictions  may  be  based  on  computer  models,  engineering 
principles,  or  engineering  judgments. 

m.  Test  Issues.  Questions  which  must  be  answered  in 
operational  and  technical  testing.  Test  issues  are  not  necessarily 
stated  in  the  same  form  as  the  system  evaluation  issues  or  system 
test  and  evaluation  critical  issues  from  which  they  are  derived, 
but  test  issues  must  be  stated  in  a  manner  that  ensures  those 
evaluation  issues  amenable  to  test  can  be  answered.  The  emphasis 
of  test  issues  is  on  producing  data  in  support  of  the  operational 
and  technical  evaluations.  Test  issues  have  criteria  when  needed. 
Test  issues  and  their  criteria  are  identified  by  the  independent 
evaluators  and  published  in  Independent  Evaluation  Plans  (lEPs)  and 
Test  Design  Plans  (TDPs) . 

n.  Conventional  Weapon.  Those  weapons  which  are  neither 
nuclear,  chemical,  or  biological. 


Section  VII 
Keys  to  Success 


1-14 .  Keys  to  Success 

The  LFT&E  program  has  and  will  continue  to  be  one  of  the  most 
complex  and  high-visibility  phases  of  weapon  system  development. 

It  requires  proper  planning,  resourcing,  testing,  evaluation,  and 
coordination  to  ensure  that  critical  vulnerability/  lethality 
issues  are  effectively  and  adequately  addressed  and  that  the 
Congressional  mandate  is  satisfied.  Based  on  the  experience  gained 
during  previous  Army  LFT  Programs  (Bradley  and  Abrams) ,  a  number  of 
"keys  to  success"  have  been  identified  which  should  be  useful  for 
future  LFT&E  programs.  These  "keys"  include: 

a.  Integration  into  the  Test  and  Evaluation  (T&E)  Process. 

The  requirements  of  LFT&E  are  comparable  to  those  of  any  T&E 
program:  one  must  identify  the  critical  issues,  develop  a  test 
strategy,  coordinate  and  obtain  approval  of  that  strategy,  and 
execute  and  report  the  results  of  that  program.  Thus,  the  existing 
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T&E  process  not  only  provides  an  existing  infrastructure  and 
reporting  system  which  can  effectively  and  efficiently  accommodate 
the  requirements  of  LFT&E,  but  it  also  avoids  the  unnecessary  step 
of  establishing  a  separate  and  unique  process  simply  for  LFT&E.  As 
will  be  discussed  in  latter  sections  of  this  guide,  the  TEMP  is 
ideally  suited  for  articulating  the  LFT&E  strategy  and  the  Test 
Integration  Working  Group  (TIWG)  is  an  ideal  forum  for  the 
planning,  coordination,  and  integration  of  the  LFT&E  program. 

b.  Early  Planning.  The  resource  demands,  plus  the  review  and 
approval  process,  for  LFT&E  make  early  planning  absolutely 
essential.  Early  identification  of  the  critical  vulnerability 
and/or  lethality  issues,  the  LFT&E  strategy,  the  test  resource 
requirements,  test  limitations,  and  inclusion  in  the  TEMP  are 
necessary  to  provide: 

(1)  The  HQDA/OSD  an  understanding  of  the  basic  strategy  and 
the  adequacy  of  planned  testing  and  resources. 

(2)  The  Project  Manager  (PM)  an  understanding  of  the  system 
hardware  and  threat/threat  surrogate  requirements,  many  of  which 
require  long  lead  times  to  procure/develop. 

c.  Building-Block  Approach.  The  key  to  understanding  a  given 
munition/ target  interaction  is  an  understanding  of  the  underlying 
phenomenology.  These  insights  can  often  be  gained  and  many 
critical  issues  addressed  through  component  and/or  sub-system 
testing.  Thus,  the  most  cost  effective  and  efficient  approach  to 
LFT  is  a  building-block  approach.  Using  such  an  approach,  a 
development  program  would  progress  from  early  component  testing  to 
sxib-sy stem/ system  testing  and  culminate  in  a  limited  series  of 
fuXl-up  firings  to  address  personnel  casualty,  damage  mechanism, 
and  critical  system  vulnerability/ lethality  issues  which  can  only 
be  answered  through  full-up  testing.  The  building-block  approach 
provides  the  earliest  possible  understanding  of  the  munition/target 
interaction  pher  ena  during  the  development  process  and  enables 
required  fixes  .  identified  problems  to  be  incorporated  at  the 
earliest  possibl*  date. 

d.  Matrix  Team  Approach.  The  complexity  of  LFT&E  programs 
requires  that  a  broad  range  of  technical,  programmatic,  and 
management  expertise  be  brought  together  for  the  planning, 
execution,  and  reporting  of  that  program.  A  matrix  team  approach 
has  been  found  to  be  the  most  effective  and  efficient  approach  in 
previous  LFT&E  efforts  for  bringing  this  diverse  set  of  expertise 
and  activities  together  and  ensuring  a  coordinated  and  credible 
LFT&E  program.  Thus,  successful  execution  of  a  LFT&E  program 
demands  the  early  recognition  of  the  need  for,  the  solicitation  of, 
the  support  of,  and  the  continuous  in\ olvement  of  all  necessary 
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activities.  Principal  team  members  typically  include  the  system 
developer,  combat  developer,  independent  evaluators, 
vulnerability/ lethality  analysts,  testers,  medical  community, 
intelligence  community,  and  system  contractor  (as  required) . 
Generally,  this  matrix  team  will  remain  in  existence  throughout  the 
LFT&E  program  and  should  be  organized  as  a  separate  working  group 
under  the  TIWG.  Membership  may  be  expanded  or  modified  as  required 
and  as  the  program  evolves. 

e.  Test  Discipline.  Strict  discipline  is  required  during  the 
test  conduct  to  ensure  validity  of  results  and  efficient  test 
execution.  This  discipline  includes  strict  adherence  to  the  HQDA 
approved  Detailed  Test  Plan  (DTP) ,  approval  of  DTP  changes  by  HQDA, 
controlled  access  to  the  test  item,  and  early  reporting  of  emerging 
results.  Test  discipline  is  discussed  in  greater  detail  in 
Chapter  5. 


Section  VIII 
Roles 


1-16.  Office  of  the  Secretary  of  Defense  (OSD) 

a.  The  Deputy  Director,  Defense  Research  and  Engineering  (Test 
and  Evaluation)  (DDDRE(T&E) ) ; 

(1)  Serves  as  the  OSD  focal  point  for  review,  coordination, 
and  approval  of  LFT&E  policy. 

(2)  Approves  LFT&E  strategies,  in  coordination  with  the 
Director  of  Operational  Test  and  Evaluation,  as  provided  in  the 
TEMP  and  in  accordance  with  (lAW)  DoD  5000. 2-M. 

b.  Director,  Live  Fire  Testing  (DLFT) ,  Office  of  the  Deputy 
Director,  Defense  Research  and  Engineering  (Test  and  Evaluation) : 

(1)  Ensures  that  the  Services  implement  all  aspects  of  the 
legislation  covering  LFT. 

(2)  Develops,  recommends,  and  supervises  DoD  LFT&E  policy. 

(3)  Reviews  and  recommends  approval  of  Service  LFT&E 
strategies  as  provided  in  the  TEMP  and  Service  proposed  deviations 
to  the  approved  LFT&E  strategies. 

(4)  Reviews  and  comments  upon  Services'  Detailed  LFT&E 
Plans  and  LFT&E  Reports. 
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(5)  Monitors  the  Services'  LFT&E  program  during  its 
conduct . 

(6)  Conducts  an  independent  assessment  of  individual 
Services'  LFT&E  programs  and  prepares  the  Secretary  of  Defense 
LFT&E  Report  to  Congress. 

(7)  Advocates  the  development  of  improved  instrumentation, 
methodologies,  criteria,  and  facilities  for  conducting  LFT&E. 

(8)  Provides  a  focal  point  for  the  identification  of 
reguirements  for  foreign  targets  and  ammunition  for  LFT. 

1-17.  Headquarters,  Department  of  the  Army  (HQDA) 

a.  The  Deputy  Under  Secretary  of  the  Army  for  Operations 
Research  (DUSA(OR)): 

(1)  Serves  as  the  HQDA  focal  point  for  review, 
coordination,  and  approval  of  Army  LFT&E  policy. 

(2)  Identifies  to  OSD  Army  systems  with  a  LFT  requirement. 

(3)  Serves  as  the  Army  approval  authority  of  LFT&E 
strategies  as  provided  in  the  TEMP  and  lAW  DoD  5000. 2-M. 

(4)  Approves  LFT&E  lEPs,  TDPs,  DTPS,  and  Detailed  Test 
Reports  (DTRs) ;  reviews  LFT&E  Independent  Evaluation  Reports 
(lERs) . 

(5)  Approves  any  deviations  to  approved  DTPS  and  lEPs. 

(6)  Authorizes  and  coordinates  the  transfer  of  validated 
LFT  data  to  the  DLFT  or  his  designated  representative (s)  on  a 
mutually  agreed  upon  schedule.  (Validated  data  are  raw  data  which 
have  been  subjected  to  a  quality  control  review) . 

b.  The  Director  for  Program  and  Vulnerability  Assessment, 
Office  of  the  Assistant  Secretary  of  the  Army  (Research, 
Development,  and  Acquisition) : 

(1)  Provides  the  Army  Acquisition  Executive  (AAE) ,  Army 
System  Acquisition  Review  Council  (ASARC)  Members,  and  Program 
Executive  Officers  (PEOs) /PMs  results  of  independent  assessments  of 
analytical,  test  and  evaluation,  countermeasures  (CM) ,  counter¬ 
countermeasures  (CCM) ,  and  vulnerability  (including  LFT&E)  issues 
on  programs  before  all  milestone  decision  reviews. 

(2)  Provides  guidance,  policy,  and  direction  with  respect 
to  CM/CCM,  vulnerability,  and  survivability  for  all  AAE  programs. 
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(3)  Oversees  vulnerability  programs  throughout  Army. 

c.  Deputy  Chief  of  Staff  for  Intelligence  (DCSINT) : 

(1)  Ensures  Defense  Intelligence  Agency  (DIA)  approved 
threat  characteristics  are  provided  to  support  design,  development, 
and  validation  of  threat  surrogates. 

(2)  Approves  threat  surrogates. 

d.  Program  Manager  (PM) : 

(1)  Informs  DUSA(OR),  through  TEMA,  of  system  LFT&E 
requirement.  If  system  does  not  have  LFT&E  requirement,  PM  so 
identifies  in  the  TEMP. 

(2)  Provides  membership  to  the  LFT&E  working  group. 

(3)  Provides  required  resources  (funding,  to  include  that 
required  for  acquisition  of  targets  and  threat  ammunition;  spare 
parts ;  etc . ) . 

(4)  Recommends,  during  LFT&E  vulnerability  testing,  whether 
shots  deemed  catastrophic  should  be  conceded. 

(5)  Provides  required  information  on  system  configuration. 

(6)  Provides  system  contractor  support,  as  required. 

(7)  Assures  that  all  User  directed  design  fixes  identified 
during  LFT&E  are,  within  program  constraints,  developed  and 
implemented. 

1-18.  Army  Materiel  Command  (AMC) 

a.  Headquarters,  AMC: 

(1)  Provides  LFT&E  oversight  and  ensures  support  of  AMC 
activities. 

(2)  Ensures  LFT&E  guidance  is  staffed  and  incorporated  in 
appropriate  Army  policy  and  procedural  documents. 

b.  Army  Materiel  Systems  Analysis  Activity  (AMSAA) : 

(1)  Forms  and  leads  the  LFT&E  working  group  under  the  TIWG. 

(2)  Serves  as  lead  organization  for  the  development  of  the 
LFT&E  strategy  and  the  preparation  of  the  TEMP  section. 
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(3)  Identifies  and  defines  critical  vulnerability  and/or 
lethality  issues  and  ensures  issues  are  incorporated  in  the  TEMP 
and  the  LFT&E  lEP/TDP. 

(4)  Develops  LFT&E  lEP/TDP. 

(5)  Develops  LFT&E  lER  as  a  separate  stand-alone  companion 
document  to  the  DTR.  Ensures  lER  is  completed  in  a  timely  manner 
to  meet  LFT&E  milestone  requirements. 

(6)  Defines  threat  surrogate  requirements,  coordinates 
these  with  the  LFT&E  working  group,  and  provides  a  list  of  proposed 
surrogates  to  the  intelligence  community  for  their  approval. 

c.  U.S.  Army  Research  Laboratory  (ARL) : 

(1)  Survivability /lethality  Analysis  Directorate ( SLAD) : 

(a)  Serves  as  the  principal  activity  in  the  Army  for 
determining  the  survivability/lethality  and  vulnerability  (SLV)  of 
Army  systems  to  the  full  spectrum  of  battlefield  threats. 

(b)  Acts  as  the  Army  focal  point  for  technical  advice 
and  consultation  on  vulnerability  and  lethality  matters  for 
decision  makers,  system  managers  and  developers,  users,  testers, 
independent  evaluators  and  other  SLV  customers. 

(c)  Provides  independent  technical  judgements  on  complex 
technical  issues  regarding  the  SLV  of  Army  systems. 

(2)  SLAD  Integration  Office; 

(a)  Serves  as  the  AMC  spokesman  on  SLV  for  the  Director 
at  major  milestone  decision  points. 

(b)  Ensures  appropriate  AMC  support  i  provided  to 

vulnerability /lethality  assessments  and  LFT&E  p:  rams. 

(c)  Provides  membership  to  the  LFT&E  working  group. 

(d)  Ensures  data  and  lessons  learned  from  LFT&E  are 
incorporated  into  vulnerability/lethality  assessment  techniques  and 
supporting  data  bases. 

(3)  Ballistic  Vulnerability /Lethality  Division  (BVLD) : 

(a)  Serves  as  the  Army  lead  laboratory  for  conventional 
ballistic  vulnerability/lethality  assessments. 
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(b)  Provides  membership  to  LFT&E  working  group. 

(c)  Assists  AMSAA  in  identification  of  critical 
vulnerability /lethality  issues  and  in  the  development  of  the  test 
design  and  data  requirements. 

(d)  Develops  and  improves  vulnerability/ lethality 
assessment  techniques  to  include  incorporation  of  LFT&E  lessons 
learned  in  assessment  techniques  and  supporting  data  bases. 

(e)  Conducts  vulnerability/ lethality  assessments  for 
decision  reviews;  provides  pre-shot  predictions/  assessments  for 
LFT&E.  Prepares  Pre-Shot  Prediction  Report. 

(f)  Leads  crew  casualty  and  system  damage  assessments. 
Prepares  Detailed  Damage  Assessment  Report. 

d.  U.S.  Army  Test  and  Evaluation  Command  (TECOM) ; 

( 1 )  Headquarters 

(a)  Plans,  coordinates,  and  manages  the  execution  of 

LFTs. 

(b)  Provides  membership  to  the  LFT&E  working  group. 

(c)  Assists  AMSAA  in  the  development  of  the  LFT&E 

strategy . 

(d)  Manages  preparation  of  the  DTP  and  the  DTR  for  those 
LFTs  assigned  for  execution. 

(e)  Reviews  the  DTP  and  DTR  for  those  LFTs  assigned  to 
other  agencies  for  execution. 

(2)  Combat  Systems  Test  Activity  (CSTA) : 

(a)  Serves  as  the  TECOM  Center  of  Excellence  for  LFT. 

(b)  For  assigned  tests: 

1  Plans  and  conducts  all  required  test  efforts  including 
instrumentation,  execution,  target  repair,  and  maintenance. 

2.  Serves  as  lead  for  preparation  of  DTP  and  DTR. 

1  Documents  test  results  and  supports  damage  assessment 
process . 
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(c)  Monitors  tests  conducted  at  other  installations. 


e.  Project  Manager  for  Instrumentation,  Targets  and  Threat 
Simulators  (PM  ITTS) : 

(1)  Implements  Army  and  AMC  policy  for  the  management, 
control,  operation  and  support  (O&S)  of  foreign  assets  to  include 
those  used  in  support  of  LFT&E. 

(2)  Interfaces  with  other  DoD  agencies  and  enters  into 
Memorandum  of  Agreement /Under standing  (MOA/MOU) ,  as  necessary,  for 
the  O&S  of  foreign  assets  to  include  those  used  to  support  LFT&E. 

(3)  Assess  requirements  for  the  use  of  foreign  assets  to 
support  LFT&E  and  forwards  recommendations  and/or  requests  for 
destructive  testing  to  the  Army  Foreign  Materiel  Review  Board. 

1-19.  Other  Army  Components 

a.  Training  and  Doctrine  Command  (TRADOC)  (Centers  and 
Schools) : 

(1)  Serves  as  lead  for  the  Battlefield  Damage  Assessment 
and  Repair  (BDAR)  team. 

(2)  Participates  in  preparation  of  DTR. 

(3)  In  coordination  with  PM,  identifies  fixes  that  result 
from  LFT&E  and  establishes  implementation  priority. 

b.  Medical  Research  and  Development  Command: 

(1)  Assists  in  the  development  of  LFT  plans  to  ensure 
medically  relevant  issue?;  are  addressed. 

(2)  Assists  in  .  itification  of  critical  crew 
vulnerability  issues  and  in  the  development  of  the  criteria  for 
casualty  assessments. 

(3)  Develops  and  improves  crew  vulnerability  assessment 
techniques  to  include  incorporation  of  LFT&E  lessons  learned  in 
assessment  techniques  and  supporting  data  bases. 

(4)  Provides  assistance  as  required  in  crew  casualty 
assessments;  reviews,  in  a  timely  manner,  the  final  casualty 
assessments  to  ensure  medically  relevant  concerns  have  been 
adequately  addressed. 
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c.  Operational  Test  and  Evaluation  Command  (OPTEC) : 
required,  provides  membership  to  the  LFT&E  working  group. 


As 
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Chapter  2 
LFT&E  Strategy 


Section  Z 
Introduction 

2-1.  General 

a.  The  development  and  subsequent  approval  of  the  LFT&E 
strategy  is  the  single  most  important  step  in  the  overall  LFT&E 
process.  The  LFT&E  strategy  is  a  documented  concept  that  describes 
who,  what,  why,  when,  where,  and  how  the  LFT&E  requirements  for  a 
given  system  will  be  satisfied.  Just  as  a  system's  acquisition 
strategy  outlines  the  top  level  approach  for  the  overall  system 
acquisition,  the  LFT&E  strategy  provides  the  top  level  description 
of  the  LFT&E  portion  of  the  system  acquisition  and  is  an  integral 
part  of  the  TEMP.  Once  approved,  the  LFT&E  strategy  provides  the 
basic  roadmap  for  what  vulnerability/ lethality  testing  and 
evaluation  has  to  be  conducted  prior  to  transitioning  to  full-rate 
production. 

b.  While  the  details  of  the  LFT&E  strategy  will  vary  from 
system-to-system,  this  Part  attempts  to  provide  the  general  details 
necessary  for  the  development  of  an  adequate  and  credible  LFT&E 
strategy.  Development  of  the  LFT&E  strategy  requires  both  an 
understanding  of  the  system's  acquisition  strategy  and  the  overall 
T&E  process.  An  overview  of  the  T&E  process  is  provided  in  Part 
One. 


Section  II. 

LFT&E  in  the  Accpiisition  Process 
2-2.  Materiel  Acquisition  Process 

Figure  2-1  depicts  where  the  elements  of  the  required 
vulnerability/ lethality  assessment  and  the  LFT&E  program  fall 
within  the  materiel  acquisition  process  as  outlined  in 
DoDI  5000.2.  Table  2-1  presents  an  outline  schedule  of  LFT&E 
events  which,  if  followed,  will  result  in  a  timely  and  effectively 
executed  LFT&E  program.  The  schedule  for  the  DTP,  DTR,  and  lER  are 
OSD  mandated  requirements  . 


(Insert  Figure  2-1) 


Table  2-1.  LFT&E  Schedule 


Schedule 

LFT&E  Event 

Lead 

Pre-MS  I 

LFT&E  Working  Group 

Formation 

AMSAA 

MS  I 

Initial  LFT&E 

Strategy 

AMSAA 

TEMP  Input; 

Resources 

PM 

MS  II 

Detailed  LFT&E 

Strategy 

AMSAA 

TEMP  Input: 

Resources 

PM 

T-180 

LFT&E  lEP/TDP 

AMSAA 

T-60 

Army  Approved  LFT&E  lEP/TDP,  DTP, 
and  Pre-Shot  Prediction  Report  to  OSD 

DUSA(OR) 

T 

LFT 

Tester 

T+60 

LFT&E  DTR 

TECOM 

T+110 

LFT&E  lER 

AMSAA 

T+120 

LFT&E  DTR  and  lER  to  OSD 

DUS A (OR) 

T+180 

Detailed  Damage  Assessment  Report 

SLAD/BVLD 

MS  =  Milestone 

T-i  or  T+i  are  documentation  time  requirements  (in  i  days)  relative 
to  the  initiation  (T-i)  or  completion  (T+i)  of  LFT,  respectively. 
The  Detailed  Damage  Assessment  Report  is  not  required  prior  to  the 
full-rate  production  decision. 


Section  III 

LFTtE  in  the  TtE  Process 
2-3.  General 

Live  Fire  Tests  are  part  of  developmental  tests  of  system 
vulnerability  and  lethality.  What  has  changed  from  previous 
developmental  tests  is  that  a  more  comprehensive  full-up  system 
test  with  OSD  oversight  is  required  before  a  program  may  enter 
full-rate  production.  The  LFT&E  program  examines  the  full  spectrum 
of  battlefield  threats,  to  include  overmatching  threats,  as  opposed 
to  the  design  level  threats  considered  in  previous  developmental 
tests.  The  scope  of  LFT&E  should  build  upon  early  developmental 
tests  of  component  and  system  vulnerability  and  lethality  and 
modeling.  Resource  and  schedule  constraints  and  the  stochastic 
nature  of  LFTs  limit  the  scope  of  these  tests  to  a  demonstration  of 
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system  vulnerability  and  lethality. 

2-4.  Factors  Addressed  in  TtE 

System  developmental  tests  and  evaluations  typically  address  the 
following  factors:  firepower  (lethality  is  an  element) ; 
survivability  (vulnerability  is  an  element) ;  performance; 
reliability,  availability,  maintainability,  and  durability; 
manpower  and  personnel  integration;  integrated  logistics  support; 
and  software.  The  LFT&E  program  addresses  elements  of  firepower 
and  survivability;  firepower  and  survivability  are  compared/ 
contrasted  in  Table  2-2. 


Table  2-2.  Elements  of  Firepower  and  Survivability 

Firepower  Survivability 


•  Ability  to  acquire  targets 

•  Ability  to  hit  an  acquired 

target 

•  AbWty  to  kU  a  target  given  a  Mt 

(lethality) 


••  Ability  to  perforate  or 
breach  target 


•  •  Ability  to  do  significant 
damage  to  the  target 


•  Rate  of  aimed  fire 


•  Avoid  or  reduce  acquisition 

•  Avoid  or  reduce  being  hit 

given  an  acquisition 

•  A  void  or  reduce  being  killed  given 
a  hit  (vulnerability) 


•  •  Protect  against  lethal 
mechanisms 


•  •  Limit  damage  to  crew 
and  hardware 


•  Design  for  expedient  repair  of 
combat  damage 


Italicized  entxxes  are  focus  of  LFT&E. 
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2-5.  System  Performance 

Both  lethality  and  vulnerability  LFT&E  address  system  performance 
given  a  munition  impact.  At  the  sub-element  level,  lethality  LFT&E 
addresses  both  the  ability  to  perforate  or  breach  the  target  and  to 
do  significant  damage  to  the  target.  Vulnerability  LFT&E  addresses 
both  being  protected  against  lethal  mechanisms  and  minimizing 
damage  to  the  crew  and  hardware  given  an  impact  or  breach  by  a 
lethal  mechanism.  In  addition,  vulnerability  LFT&E  addresses 
repairability  of  combat  damage  (another  element  of  survivability) . 

2-6.  Differences  between  Vulnerability  and  Lethality  LFT&E 

There  are  several  subtle  differences  in  vulnerability/  lethality 
LFT&E.  Vulnerability  LFT&E  must  address  crew,  hardware  (excluding 
crew) ,  and  system  (crew  and  hardware)  vulnerability  for  threats  and 
impact  conditions  that  the  system  is  designed  to  protect  against 
and  for  threats  and  impact  conditions  that  the  system  is  not 
designed  to  protect  against  but  could  encounter  on  the  battlefield. 
In  lethality  LFT&E,  it  is  sufficient  to  address  lethality  against 
the  threat  system  for  areas  that  have  the  greatest  protection 
and/or  where  differences  between  competing  munitions  are  expected 
(not  only  areas  of  greatest  protection) .  For  example,  a  new 
munition  may  not  be  able  to  breach  the  area  of  greatest  protection 
on  the  threat;  however,  for  areas  that  it  can  breach,  the  damaging 
effects  (probability  of  kill  given  a  hit  (Pk/t) )  may  be 
significantly  greater  than  the  munition  being  replaced. 


Section  IV 

Developing  the  LFT6E  Strategy 
2-7.  General 

The  LFT&E  strategy  is  the  most  important  element  of  the  LFT&E 
process.  It  should  be  prepared  and  approved  as  early  as  possible 
in  the  acquisition  cycle  (see  Table  2-1) .  The  AMSAA  has  the  lead 
for  preparing  and  obtaining  approval  for  the  strategy  in 
coordination  with  TIWG  members.  Thr  ’SA(OR)  approves  the  strategy 
for  the  Army  before  it  is  sent  (via  e  TEMP)  to  the  DLFT  for  OSD 
approval.  If  consensus  on  the  scope  of  the  LFT&E  cannot  be 
reached,  or  if  program  constraints  limit  compliance  with  required 
reporting  dates,  these  issues  will  be  raised  to  the  DUSA(OR)  for 
resolution.  The  strategy  is  the  foundation  of  the  LFT&E  section  of 
the  TEMP  and  all  subsequent  planning  documents  (the  lEP/TDP 
prepared  by  AMSAA,  the  Pre-Shot  Prediction  Report  prepared  by 
SLAD/BVLD,  and  the  DTP  whose  preparation  is  managed  by  TECOM) .  The 
strategy  should  be  detailed  enough  to  adequately  project  resource 
requirements  and  trigger  long  lead  time  planning ,  procurement  of 
threats /surrogates,  and  modeling. 
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2-8.  Steps  to  take 

The  first  step  in  preparing  a  strategy  is  to  do  the  necessary 
homework  to; 

a.  Understand  the  technical  and  operational  characteristics  of 
the  concepts,  technology,  and  requirements  for  the  system  being 
developed  and  how  they  differ  from  the  system  being  replaced. 

b.  Develop  a  rationale  for  which  threats  are  to  be  considered 
in  the  LFT&E.  The  rationale  should  be  based  upon  a  review  of  the 
STAR,  the  densities  of  the  various  classes  of  threat  weapons  in 
orqanizations  likely  to  be  encountered,  and  the  frequency  that 
various  threats  kill  or  are  killed  by  the  system  from  force 
effectiveness  analyses  supporting  program  decisions  or  planning 
studies.  An  accepted  rationale  from  an  approved  vulnerability 
LFT&E  plan  was  to  break  threats  into  major  and  minor  threats,  a 
major  threat  was  either  one  that  killed  a  large  percentage  of  the 
systems  in  the  force  effectiveness  evaluation  or  had  a  high  density 
in  the  force;  all  others  were  considered  minor  threats.  Most  or 
the  shots  fired  in  vulnerability  LFT&E  should  be  with  ma^or 
threats.  The  rationale  for  lethality  LFT&E  should  be  based  on  the 
threat  that  is  driving  the  design  (usually  the  most  difficult 
target  to  kill  given  a  hit) . 

c  Review  previous  LFT  or  JLF  results  for  the  system  being 
replaced.  If  vulnerability  tests  have  been  conducted,  repeating 
some  shots  with  the  system  being  developed  may  significantly  reduce 
or  eliminate  the  need  for  comparison  tests  with  the  system  being 
replaced.  Previous  LFT  and  JLF  tests  may  have  identified 
vulnerability  issues  that  need  further  exploration  or  designs  that 
could  reduce  system  vulnerability.  This  should  influence  the  scope 
of  the  vulnerability  LFT.  For  example,  during  developmental 
testing  of  the  MlAl  Abrams  Tank,  damage  caused  by  ballistic  shock 
from  nonperforating  impacts  was  identified  as  a  potentially 
significant  vulnerability  of  the  tank.  The  Vice  Chief  of  Staff  of 
the  Army  directed  a  ballistic  shock  test  of  the  Ml  tanks  in 
production.  That  test  identified  design  fixes  that  were 
incorporated  into  the  MlAl  design.  The  Abrams  Vulnerability  LFT 
evaluated  how  well  the  design  modifications  worked. 

d.  Identify,  for  lethality  LFT&E,  threat  target  requirements 
and  availability.  The  PMs  are  responsible  for  funding  the 
acquisition  of  targets  for  lethality  LFT&E  (see  Section  V  and 
Chapter  10  for  threat  target  alternatives) .  In  the  past,  JLF 
targets  have  been  made  available  to  support  LFT&E  testing. 
Providing  developmental  rounds  to  fire  in  JLF  tests  may  satisfy 
some  or  all  of  the  full-up  threat  target  portion  of  the  lethality 
LFT&E  requirement.  In  addition,  it  may  be  possible  to  infer  that 
the  developmental  round  would  be  at  least  as  lethal  as  similar 
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(less  capable)  rounds  fired  in  the  test.  This  could  be  used  as  a 
justification  for  firing  fewer  developmental  rounds  in  LFT&E.  An 
additional  potential  source  of  threat  targets  for  use  in  LFT&E  is 
PM  ITTS,  the  AMC  management  agent  of  foreign  materiel  assets  used 
in  support  of  testing.  (See  Part  8  for  procedures  for  the 
acquisition  of  targets  through  PM  ITTS  to  support  LFT&E.) 

2-9.  Critical  Issues 

Having  completed  the  homework  on  the  developmental  system,  the  next 
step  in  developing  a  strategy  is  to  define  the  critical  issues 
(test  issues) .  Critical  issues  are  not  unique  to  the  LFT&E  phase, 
but  are  issues  which  are  developed  to  address  overall  system 
vulnerability  and/or  lethality,  (i.e.,  they  are 

vulnerability/ lethality  critical  issues) .  The  LFT&E  program  will 
address  specific  elements  of  the  overall  system 
vulnerability/ lethality  issues.  Critical  issues  vary  for 
vulnerability  and  lethality  and  generally  should  address  the 
following; 

a.  Vulnerability  LFT&E; 

(1)  Crew,  hardware,  and  system  vulnerability. 

(2)  Known  vulnerabilities  and  vulnerability  reduction 
techniques  (e.g.,  increased  ballistic  protection,  less  sensitive 
munitions,  and  redundant  components) . 

(3)  Potential  vulnerability  reduction  techniques. 

(4)  Processes,  provisioning,  repair  times,  and  training 
required  for  BDAR. 

Testing  should  also  provide  valuable  inputs  and  a  basis  for 
refinement  and  calibration  of  vulnerability  and  Sustainability 
Predictions  for  Army  Requirements  for  Combat  (SPARC)  models. 

b.  Lethality  LFT&E; 

(1)  Ability  to  perforate  or  breach  the  protection  of  the  threat 
system. 

(2)  Ability  to  significantly  degrade  the  combat/mission 
functions  of  threat  systems  given  a  breach. 

(3)  Potential  lethality  improvements. 

Testing  should  also  provide  valuable  inputs  and  a  basis  for 
refinement  and  calibration  of  lethality  models  and  data  bases. 
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2-10.  LFTSE  lEP/TDP  ^  . 

During  the  examination  of  the  vulnerability/ lethality  of  the  system 
being  developed  and  the  defining  of  the  critical  issues,  the 
process  by  which  the  LFT&E  results  will  be  evaluated  is  formulated. 
The  next  step  in  the  strategy  development  is  the  finalization  of 
the  evaluation  process  and  the  articulation  of  the  details  of 
process  in  the  LFT&E  lEP/TDP  document.  This  document  will  identify 
procedures  to  be  followed  in  the  evaluation  (e.g.,  statistical 
analyses,  criteria,  models,  system  comparisons,  shotlines)  and 
define  data  requirements.  During  development  of  the  LFT&E  strategy 
and  the  resultant  lEP/TDP,  the  total  vulnerability/ lethality 
assessment  process  must  be  considered.  The  evaluation  must 
crosswalk  the  developmental,  component,  sub-system,  etc., 
vulnerability/lethality  testing  and  assessments  with  LFT&E 
requirements.  Some  aspects  of  the  assessment  process  which  must  be 
examined  in  the  development  of  the  LFT&E  strategy  are; 

a.  Early  in  the  system  acquisition  cycle  there  is  little  or  no 
test  data  and  the  evaluations  are  made  based  upon  model  estimates. 
Data  bases  to  support  the  models  should  reflect  the  technical  and 
performance  characteristics  of  the  system  and  the  threat. 

initial  models  and  model  inputs  will  probably  be  both  unrefined  and 
uncertain.  The  LFT&E  strategy  should  be  designed  to  increase  the 
level  of  refinement  and  to  reduce  the  uncertainty. 

b.  Models  for  both  vulnerability  and  lethality  evaluations 
require  similar  inputs.  A  detailed  description  of  the  system  is 
required  for  a  vulnerability  assessment.  A  detailed  description  of 
the  threat  target  is  required  for  a  lethality  assessment.  These 
descriptions  must  geometrically  describe  the  location  of  the 
critical  components,  crew,  and  protective  system.  In  addition,  one 
or  two  other  sets  of  data  is  required:  a  damage  assessment  list 
(DAL) ,  or  a  degraded  states  analysis.  The  DAL  provides  the 
expected  loss  of  combat  function  (e.g.,  mobility  or  firepower) 
associated  with  damage  to  components  and  crew.  The  DAL  must  be 
developed  by  vulnerability  analysts  and  system  users.  A  degraded 
states  analysis  relates  damage  to  components  and  crew  to  an 
expected  loss  of  system  capability.  (Unlike  the  DAL  process, 
degraded  states  analysis  makes  no  attempt  to  relate  loss  of 
capability  to  some  combat  utility,  thus  avoiding  averaging  over 
some  spectrum  of  mission  scenarios.  The  only  difference  between 
degraded  states  and  the  DAL  is  that  degraded  states  analysis  allows 
the  user  to  apply  his  own  mission  profile,  rather  than  using  the 
one  implicit  in  the  DAL).  Finally,  the  ability  of  the  system's 
pjfotective  system  to  withstand  an  impact^ by  the  threat,  the 
characterization  of  the  damaging  capability  of  fhe  threat  that 
breaches  the  protective  system;  and  the  susceptibility  of  the 
components  and  crew  to  the  threat  damage  mechanisms  are  required. 
Comparable  information  is  required  on  threat  targets  for  lethality 
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evaluations. 

c.  Vulnerability  developmental  tests  must  be  planned  to  assess 
the  ability  of  a  system's  protective  system  armor)  to 

withstand  impacts  by  threat  missiles  and  projectiles  and  to  examine 
the  ability  of  critical  components  (e.g.,  ammunition  compartments) 
to  withstand  damage  from  a  threat  warhead  or  projectile  that 
breaches  the  protective  system.  During  the  Demonstration  and 
Validation  Phase,  the  developmental  tests  will  focus  on  components. 
During  EMD  and  production  qualification  testing  (PQT) ,  complete 
systems  should  be  tested;  however,  developmental  tests  should  be 
planned  to  upgrade  and  develop  the  system  vulnerability  model.  The 
vulnerability  LFT  is  the  last  vulnerability  developmental  test  and 
should  be  conducted  against  a  full-up  (combat- loaded)  production 
system. 


d.  Lethality  developmental  tests  must  be  planned  to  assess  the 
ability  of  the  system  to  damage  critical  components  and  the  crew. 
During  the  Demonstration  and  Validation  Phase,  the  tests  will 
usually  focus  on  the  warhead's  or  penetrator's  ability  to  breach 
the  threat  target's  protective  system.  During  EMD  and  PQT,  impact 
conditions  will  be  firmly  established  for  the  missile  or  projectile 
and  the  ability  of  the  warhead  or  penetrator  to  breach  the  threat 
targets  protective  system  will  be  refined.  The  lethality  LFT  is 
again  the  last  developmental  test  and  should  be  conducted  against  a 
full-up  (combat-loaded)  threat  target.  It  is  unlikely  that  the 
required  threat  target  will  be  available.  (The  Army  develops 
munitions/missiles  to  "defeat"  projected  threats  which  in  most 
cases  have  not  been  fielded) .  Therefore,  surrogate  targets  will 
have  to  be  developed  or  JLF  and/or  available  threat  targets  will 
have  to  be  used.  In  either  case,  the  scarcity  of  lethality  LFT 
targets  and  their  cost  may  dictate  that  these  targets  not  be  fully 
combat-loaded  to  preclude  an  unexpected  catastrophic  loss. 

e.  Vulnerability  models  are  also  used  to  estimate  the  spare 
parts  and  time  required  to  repair  combat  damaged  components. 
Vulnerability  LFTs  provide  valuable  inputs  for  refining  these 
estimates.  In  addition,  rapidly  returning  damaged  systems  to 
battle  requires  being  able  to  accurately  assess  the  damage  and 
apply  field  expedient  repairs.  Again  vulnerability  LFTs  provide 
both  valuable  training  and  opportunities  to  refine  and  develop 
field  expedient  repair  methods  and  to  identify  tools  and  materials 
required  to  execute  these  repairs. 


Section  V 

Threat  Targets  and  Munitions 
2-11.  Requirements 
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An  integral  part  of  LFT&E  strategy  development  is  the 
identification  of  the  threat  target  (lethality  LFT)  and  munition 
(vulnerability  LFT)  requirements.  These  requirements  need  to  be 
identified  early-on  in  the  acquisition  cycle  to  allow  for  possible 
long  lead  times  for  procurement.  It  is  very  unlikely  that  the 
required  threats  will  be  available  for  LFT.  It  is  also  unlikely 
that  any  hard  intelligence  data  will  be  available  on  the  threat's 
physical  and  performance  characteristics.  Therefore,  LFTs  will 
probably  be  conducted  using  threats  based  upon  postulated 
technology  options  derived  from  intelligence  assessments.  This 
will  require  surrogates  in  lieu  of  "real"  threats.  The  rationale 
for  threat  surrogate  selection  must  be  detailed  in  the  lEP/TDP. 

2-12.  Rationale  for  Vulnerability 

The  rationale  for  selecting  surrogate  threat  projectiles  for 
vulnerability  LFTs  is  to  match  physical  and  performance 
characteristics  of  the  projected  threat.  For  kinetic  energy 
projectiles,  penetration  into  rolled  homogeneous  armor  (R^) ; 
muzzle  velocity  and  impact  velocity;  and  penetrator  material, 
length,  and  diameter  are  key  parameters.  For  shaped  charge 
warheads,  penetration  into  RHA;  impact  velocity;  and  warhead 
diameter,  explosive  type,  material,  and  diameter  are  key 
parameters.  Availability  and  cost  of  surrogate  projectiles  may 
also  drive  the  selection.  Typically,  U.S.  projectiles  and  warheads 
will  be  selected  as  surrogates.  The  U.S.  projectiles  and  warheads 
selected  as  threat  surrogates  must  be  submitted,  along  with  the 
supporting  rationale,  by  AMSAA  to  the  DCSINT,  HQDA,  for  approval. 

2-13.  Rationale  for  Lethality 

The  rationale  for  selecting  surrogate  targets  for  lethality  LFTs  is 
the  same  as  that  for  selecting  surrogate  projectiles  or  warheads. 
However,  selecting  and  obtaining  surrogate  targets  is  much  more 
difficult  and  expensive  than  selecting  and  obtaining  surrogate 
projectiles  and  warheads.  It  is  the  pacing  item  and  probably  the 
most  difficult  part  of  executing  lethality  LFT  and,  as  such,  must 
be  addressed  and  identified  early  in  the  LFT&E  planning  process. 
This  problem  was  recognized  by  the  DLFT  and  the  DUSA(OR) .  The 
AMSAA  was  requested  to  chair  an  Army  working  group  (AMSAA,  BRL, 
TECOM,  and  VLAMO)  to  develop  guidelines  for  generic  classes  of 
threat  target  surrogates  to  satisfy  long-term  requirements.  The 
Army  working  group  prepared  two  papers  that  identified  and 
evaluated  alternatives  for  threat  tank  and  helicopter  targets  for 
LFT&E.  These  papers  have  been  approved  by  the  DUSA(OR) .  Details 
can  be  found  in  Chapter  7.  In  the  following  paragraphs,  a  brief 
summary  of  each  paper  is  provided. 

NOTE:  Since  the  publication  of  these  papers  the  Army  has  established  PM  ITTS  under  AMC.  The  PM 
ITTS  function  is  to  act  as  the  AMC  management  agent  of  foreign  materiel  assets  used  in  testing.  If 
PMs  require  PM  ITTS  support,  they  must  identify  their  target  requirements  (including  LFT&E)  early  in 
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the  development  cycle  (by  MS  U. 

2-14.  Tank  Alternatives 

Eight  T&E  alternatives  were  identified  for  anti-tank  munitions  in 
terms  of  the  type  of  target  utilized  in  the  lethality  LFT,  whether 
the  target  functions  (i.e.,  mobility,  firepower,  etc.),  what  the 
test  addresses  (i.e.,  armor  perforation,  damage  mechanisms, 
components) ,  and  the  basis  for  the  overall  lethality  assessment 
(i.e.,  test,  model).  The  eight  lethality  T&E  alternatives  break 
logically  into  three  groups: 

a.  Functioning  tanks  with  an  overall  lethality  assessment 
based  upon  test  results. 

b.  Ballistic  hull  and  turret  (BH&T)  with  the  crew  or  crew  and 
components  represented  by  boxes  with  a  limited  overall  lethality 
assessment  based  upon  test  results. 

c.  A  BH&T  only  or  range  targets  with  no  overall  lethality 
assessment  based  upon  test  results. 


2-15.  Recommended  Tank  LFT&E  Approach 

^  None  of  the  alternatives  by  themselves  is  adequate  for  lethality 

LFT;  however,  it  is  possible  to  utilize  three  different  targets  to 
adequately  demonstrate  lethality  in  LFTs.  The  three  different 
targets  and  the  types  of  tests  recommended  are  as  follows: 

a.  Threat  tank  range  target  tests  with  sufficient  sample  sizes 
to  establish  (with  high  statistical  confidence)  the  ability  of  the 
baseline  and  developmental  anti-armor  munition  to  perforate  the 
range  targets  of  interest  and  to  characterize  the  behind-armor 
debris  (BAD)  characteristics  of  both  munitions. 

b.  A  BH&T  target  constructed  to  threat  armor  projections  and 
configured  with  crew  and  major  component  box  representations  to 
demonstrate  major  lethality  differences  between  baseline  and 
developmental  anti-armor  munitions. 

c.  An  older  threat  or  U.S.  tank  (without  modifications)  to 
provide  a  limited  demonstration  of  lethality  of  the  baseline  and 
developmental  munitions  against  a  functioning  vehicle.  (Note  these 
tests  may  not  demonstrate  significant  differences  because  both 
munitions  may  significantly  overmatch  these  targets.) 

2-16.  Helicopter  Alternatives 

r 

\ 

\ 
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Four  T&E  alternatives  were  identified  for  anti-helicopter 
munitions.  The  four  alternatives  for  lethality  LFTs  are: 

a.  Flyable/ functioning  helicopters  with  an  overall  lethality 
assessment  based  upon  observation  of  test  results. 

b.  Non-f lyable/functioning  helicopter  targets  with  an  overall 
lethality  assessment  based  upon  a  combination  of  modeling 
(principally  to  define  intercept  and  fuzing/detonation  points)  and 
test  data  (collection  of  damage  effects  data) . 

c.  Non-f lyable/non-functioning  helicopter  targets  with  an 
overall  lethality  assessment  based  upon  a  combination  of  modeling 
(defining  intercept  and  fuzing/detonation  points  and  damage  effects 
on  non-functioning  components)  and  test  results  (collection  of 
damage  effects  data) . 

d.  Fuselage  or  major  sub-systems  representative  of  comparable 
threat  helicopter  components  (i.e.,  engine,  rotor  system,  etc.) 
with  an  overall  lethality  assessment  based  primarily  upon  modeling. 

2-17.  Recommended  Helicopter  LFT&E  Approach 

Again  the  recommended  approach  for  lethality  LFT  of  anti-helicopter 
munitions  is  a  combination  of  two  target  types: 

a.  Non-f lyable/non-functioning  helicopters  and  fuselage  or 
major  sub-systems  built  based  upon  threat  helicopter  technical 
projections. 

b.  Flyable/ functioning  and  non-f lyable/functioning  older 
threat  or  U.S.  helicopters. 


Section  VI 

Shot  Selection  Process 


2-18.  Identification 

In  order  to  provide  the  appropriate  information  required  to  address 
critical  LFT&E  issues,  the  attack  conditions  and  the 
munition/ target  impact  location  (i.e.,  shotline)  must  be  identified 
for  each  shot.  The  shotlines  selected  and  the  rationale  for  their 
selection  must  be  included  in  the  lEP/TDP.  There  are  two  types  of 
shots:  engineering  and  random.  Engineering  shots  provide 
information  and  data  to  address  specific  vulnerability  or  lethality 
issues  for  a  specific  threat.  Random  shots  are  selected  from  the 
combat  distribution  of  impact  conditions  (direction,  location,  and 
range)  for  the  threats  of  interest.  The  minimum  number  of 
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engineering  shots  should  be  selected  first  to  address  the 
vulnerability  and/or  lethality  critical  issues.  Next,  the  number 
of  random  shots  required  for  each  threat  weapon  should  be  selected. 
Random  shots  should  be  reviewed  to  determine  if  any  engineering 
shots  are  duplicated  or  if  a  critical  issue  is  satisfied  by  a 
random  shot.  Those  engineering  shots  duplicated  by  a  random  shot 
should  be  eliminated.  Finally,  if  the  system  being  developed  is  to 
replace  an  existing  system,  shots  against  the  system  being  replaced 
should  be  added  to  enable  system  comparisons. 

2-19.  Questions  to  be  Answered 

In  order  to  select  LFT&E  shots,  the  answers  to  the  following 
questions  must  be  known: 

a.  What  are  the  characteristics  of  the  system  being  developed 
and  the  system  being  replaced? 

b.  What  are  the  differences  in  system  characteristics  that 
could  influence  vulnerability  or  lethality? 

c.  What  is  the  current  state  of  knowledge  about  system 
vulnerability  or  lethality? 

d.  What  are  the  critical  issues? 

e.  What  are  the  threats? 

f.  What  are  the  physical  and  performance  characteristics  of 
the  threats? 

g.  If  threat  systems  are  not  available,  then  what  is  the 
rationale  for  threat  surrogates? 

h.  What  vulnerability  or  lethality  technical  testing  has  been 
planned/ conducted  prior  to  LFT? 

i.  Has  JLF  or  vulnerability/ lethality  testing  been  done  on  the 
system  being  replaced? 

j.  What  are  the  program  and  test  constraints? 

k.  Has  any  high  level  guidance  been  provided? 

2-20.  Additional  Discussion 

Questions  a  through  i  have  been  discussed  previously;  question  h  is 
also  discussed  below  to  reemphasize  its  importance.  Questions  j 
and  k  will  be  discussed  briefly  before  outlining  the  parameters  to 
be  considered  in  selecting  LFT&E  shots. 
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a.  Ideally  system  program  schedules  and  funding  should  be 
developed  based  upon  detailed  LFT&E  planning;  however,  early  in  the 
acquisition  cycle,  the  level  of  planning  is  usually  unrefined  and 
decisions  are  made  that  lock  in  schedules  and  funding  levels.  The 
LFT&E  program  should  be  planned  independent  of  constraints  and  then 
efforts  must  be  made  in  developing  and  approving  the  strategy  to 
obtain  relief  from  schedule  and  resource  constraints.  The  most 
likely  outcome  of  this  process  is  compromise  and  trying  to  work  out 
strategies  that  meet  the  spirit  and  intent  of  the  law  within 
existing  or  modified  constraints. 

b.  Test  facilities  may  constrain  LFTs.  There  may  be  a  need 
for  new  facilities  or  instrumentation.  Time  and  money  may  not  be 
sufficient  to  develop  new  facilities.  In  addition,  there  may  be 
competing  demands  for  LFT  facilities  for  concurrent  system 
developments . 

c.  High  level  guidance  is  frequently  provided  on  the  number  or 
percentage  of  random  shots,  threats  to  be  included  in  the  test, 
conditions  to  be  fired,  test  design  and  statistical  tests  to  use  in 
the  evaluation  (e.g.,  pairwise  comparison  using  the  Sign  Test), 
vulnerability  or  lethality  issues  to  be  assessed,  and  test  methods. 
This  guidance  must  be  taken  into  account  explicitly  in  developing 
the  strategy.  If  the  guidance  cannot  be  accommodated  then  the 
rationale  for  not  addressing  it  must  be  presented. 

d.  The  other  major  constraints  are  the  availability  of  threat 
projectiles  for  vulnerability  tests  and  threat  targets  for 
lethality  tests.  For  developmental  systems,  it  is  almost  a 
certainty  that  threat  projectiles  and  threat  targets  will  not  be 
available  or,  if  they  are,  that  they  will  be  available  in  very 
limited  quantities.  Developing  a  rationale  for  surrogates  that  is 
practical  (in  terms  of  availability  and  cost)  is  important, 
especially  for  lethality  LFT&E. 

2-21.  Parameter  Selection 

For  each  munition/target  combination,  the  following  parameters  must 
be  selected  and  specified:  range,  angle  of  attack,  and  point  of 
impact.  For  engineering  shots,  the  procedure  for  selecting  these 
parameters  is  straightforward;  select  the  threat  and  the  required 
parameters  to  address  a  specific  vulnerability/  lethality  issue. 

For  random  shots,  the  procedure  is  based  on  random  selections  from 
"battlefield”  distributions  of  the  appropriate  parameters.  The 
Board  on  Army  Science  and  Technology  (BAST)  developed  a  methodology 
for  selecting  random  shots  for  the  Bradley  Live  Fire  Vulnerability 
Test.  The  BAST  methodology  was  revised  for  the  Abrams 
Vulnerability  LFT  to  better  distribute  the  random  shots  over  the 
entire  vehicle  when  the  sample  size  was  small.  The  revised  random 
shot  methodology  was  reviewed  and  approved  by  members  of  the  BAST. 
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This  methodology  should  be  considered  for  future  LFT&E  programs. 
The  random  sampling  parameters  for  direct  fire  threats  versus  an 
armored  target  are; 

a.  Posture  (attack  or  defense) . 

b.  Range  (based  upon  attack  or  defense  posture) . 

c.  Angle  of  attack  (stratified  into  equal  probability 
intervals  to  ensure  sampling  over  all  possible  attack  angles  with 
small  sample  sizes) . 

d.  Target  side  (left  or  right). 

e.  Hull  or  turret. 

f.  Horizontal  dispersion. 

g.  Direction  of  horizontal  dispersion  (left  or  right). 

h.  Vertical  dispersion. 

i.  Direction  of  vertical  dispersion  (up  or  down) . 

2-22.  Exclusion  Rules 

Exclusion  rules  may  also  be  established  for  rejecting  random 
shotline  draws.  Typically,  these  exclusion  rules  for  armored 
targets  reject  shots  that; 

a.  Do  not  impact  turret  or  hull  armor. 

b.  Are  a  repeat  of  another  random  shotline. 

c.  Are  a  repeat  of  a  previous  full-up  vehicle  shot. 

2-23.  Parameter  Modification 

The  sampling  parameters  for  random  shot  selection  must  be  modified 
as  a  function  of  weapon  class  (e.g.,  direct  fire  weapons,  indirect 
fire  and  top  attack  weapons,  mines) .  For  example,  none  of  the 
preceding  parameters  apply  for  pressure  activated  mines.  For 
pressure  activated  mines,  the  sampling  parameters  would  include 
right  or  left  track  and  the  location  under  the  track. 
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Chapter  3 

LFT6E  Review  and  Approval  Process 

Section  I 
Introduction 


3-1.  General  ,  .  , 

Figure  1-1  provided  an  overview  of  the  LFT&E  process  from  initial 
strategy  definition  to  the  distribution  of  the  final  test 
report (s).  Key  to  that  process  is  the  review  and/or  approval  of 
the  strategy,  T&E  plans,  and  test  reports  by  senior  decision  makers 
within  HQDA  and  OSD.  The  LFT&E  review  and  approval  process  builds 
upon  the  existing  T&E  review  and  approval  process  and  ensures  that 
the  ”chain-of -command”  is  not  only  kept  informed  of,  but,  also 
approves  all  aspects  of  the  LFT&E  program  for  a  given  system.  This 
review  and  approval  process  will  ensure  an  adeguate 
vulnerability/lethality  assessment  and  provide  the  development 
community  the  necessary  information  to  conform  to  the  latest  ^E 
ASARC  review  process  guidance,  i.e.,  pre-ASARCs  and  ASARCs  will 
include  a  briefing  covering  the  assessment  of  the  vulnerability  ana 
CM/CCM  of  the  system. 


Section  II 

Test  and  Evaluation  Master  Flan 


3**2  •  GGner&l 

a.  The  TEMP  is  the  basic  planning  document  for  all  T&E  and  is 
the  document  by  which  the  Army  formally  coordinates  and  approves 
the  LFT&E  strategy  for  a  given  system  and  communicates  that 
strategy  to  OSD.  The  preparation  and  processing  of  TEMPs  is 
conducted  under  the  auspices  of  the  TIWG.  (See  Volume  1  for 
guidance  concerning  TEMP  procedures  and  formats  to  be  followed  in 
the  TEMP  preparation) .  The  TIWG  provides  the  forum  to  affect 
coordination  and  resolve  problems  in  the  LFT&E  process.  A  separate 
LFT&E  working  group  (which  AMSAA  chairs)  under  the  TIWG  is  formed 
to  prepare  the  LFT&E  strategy  and  the  LFT&E  input  to  the  TEMP.  A 
smaller  group  combined  with  the  classified  nature  of  LFT&E  enables 
these  items  to  be  developed  in  a  more  timely  and  efficient  manner. 
Additionally,  the  LFT&E  working  group  may  assist  in  any  required 
briefings  of  the  LFT&E  strategy  to  HQDA  and  OSD. 


b.  The  TEMP  (PART  III  DEVELOPMENTAL  TEST  AND  EVALUATION, 
paragraph  d,  T.ive  Fire  Test  and  Evaluation)  shall  contain  the  LFT&E 
strategy  for  the  program  throughout  its  materiel  acquisition 
process.  The  TEMP  summarizes  what,  why,  who,  where,  when,  and  how 
the  LFT&E  issues  will  be  tested  and  evaluated.  All  LFT&E  which 


3-1 


DA  Pamphlet  73-1,  Part  Six,  16  Oct  92  (Draft) 

impacts  on  program  decisions  will  be  outlined  in  the  TEMP. 

TEMP  shows  the  relationship  of  the  LFT&E  issues  to  the  require 
technical  and  operational  characteristics;  describes  the  critical 
vulnerability/ lethality  issues  and  evaluation  criteria;  outlines 
the  planned  LFT&E;  discusses  the  amount  and  type  of  LFT&E  that  will 
be  performed  to  support  each  program  decision  point;  and  indicates 
where  schedule,  resource,  or  budget  constraints  may  impact  the 
adequacy  of  planned  LFT&E.  Specific  items  to  be  addressed  in  the 
TEMP  includes  a  description  of  the  following  items: 

(1)  The  overall  LFT&E  strategy. 

(2)  Related  prior  and  future  LFT&E  efforts. 

(3)  The  evaluation  plan. 

(4)  The  major  test  limitations. 

(5)  The  shot  selection  process. 

c.  The  primary  LFT&E  resource  requirements  should  be 
identified  and  addressed  in  the  T&E  Resource  Section  of  the  TEMP  as 
early  as  possible  (to  facilitate  budget/schedule  projections) ; 
initial  resource  requirements  should  be  identified  prior  to  the 
Milestone  I  decision.  This  will  ensure  that  adequate  time  is 
allowed  for  long  lead  items  such  as  targets  for  lethality  tests  and 
threat  munitions  for  vulnerability  tests.  Additionally,  it  ensures 
the  early  identification  and  programming  of  the  funds  required  for 
test  execution. 

3-4.  Review  and  Approval  j 

Since  the  LFT&E  strategy  is  part  of  the  TEMP,  the  review  and 
approval  process  established  for  the  TEMP  (see  Part  Two)  . 

necessarily  applies  to  the  LFT&E  strategy.  Specifically,  ^SAA,  in 
coordination  with  the  TIWG,  develops  the  LFT&E  strategy  and 
incorporates  it  into  the  TEMP.  Upon  completion  of  initial 
coordination  but  prior  to  formal  TEMP  submission  to  HQDA,  it  is 
advisable  to  brief  the  LFT&E  strategy  to  the  DUSA(OR)  to  solicit 
ini’tial  guidance/ agreement  in  principle  on  the  proposal. 

NOTE:  Any  acquisition  category  program  with  a  LFT&E  requirement  is  necessarily  on  the  OSD 
oversight  list  (even  if  Just  for  LFT&E  purposes),  and  thus  such  program  '5  TEMP  must  be  submitted  to 
HODA  for  approval  prior  to  submission  to  OSD  (see  Part  Two). 

During^the%laSning^aSd  conduct  of  a  LFT&E  program,  the  TIVG  will 
attempt  to  resolve  all  issues.  Those  issues  which  cannot  be 
resolved  by  the  TIWG  will  be  forwarded  through  the  PEO/PM  to  the 
DUSA(OR)  for  final  resolution.  In  some  cases,  issues  may  be  raised 
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during  the  conduct  of  the  LFT&E  program  which  require  off-line 
tests  or  additional  full-up  firings.  In  all  cases,  any  additional 
firings  must  be  approved  by  the  DUSA(OR). 

3-6.  Waiver 

a.  The  LFT&E  legislation  contains  a  provision  which  allows  the 
Secretary  of  Defense  to  waive  the  requirement  for  full-up  LFT&E,  if 
the  Secretary  of  Defense  certifies  to  Congress  that  such  LFT&E 
would  be  unreasonably  expensive  and  impractical.  In  time  of  war  or 
mobilization,  the  LFT&E  requirement  may  be  suspended  by  the 
President.  A  request  for  waiver  must  be  submitted  and  approved 
prior  to  the  Milestone  II  decision.  The  review  and  approval 
process  for  waivers  is  as  follows: 

(1)  The  request  for  waiver  is  prepared  by  the  PM  and  must 
include  the  strategy  which  will  be  followed  in  assessing  overall 
system  vulnerability/ lethality  in  lieu  of  full-up  testing  and  an 
assessment  of  possible  alternatives  to  realistic  system  testing. 

(2)  Request  for  waiver  is  submitted  by  the  PM  to  the  TIWG 
for  coordination  and  approval. 

(3)  Upon  TIWG  approval,  the  PEO/PM  submits  the  request  for 
waiver  through  the  DUSA(OR)  for  review  and  approval  by  the  AAE. 

(4)  Upon  approval  by  the  AAE,  the  DUSA(OR)  submits  the 
request  for  waiver  through  the  DDDRE(T&E)  to  the  Undersecretary  of 
Defense  for  Acquisition  for  approval  by  the  Secretary  of  Defense. 

b.  The  waiver  process  should  normally  be  considered  a  last 
resort  in  addressing  the  full-up  LFT&E  requirement.  The  LFT&E 
Guidelines  issued  by  OSD  and  endorsed  by  the  DUSA(OR)  provide 
sufficient  latitude  for  a  broad  range  of  systems  configurations  to 
satisfy  the  LFT&E  requirement  without  having  to  resort  to  a  request 
for  wavier.  Specifically,  the  Guidelines  allow  full-up  testing 
(full-up  components,  full-up  sub-systems,  etc.),  in  lieu  of  full- 
up,  full-scale  system  testing  in  addressing  the  LFT&E  requirement. 
The  development  and  articulation  of  a  well-planned  strategy  which 
takes  advantage  of  extensive  component/ sub-system/ system  testing 
and  a  limited  but  reasonable  full-up,  sub-system/ system  LFT&E  phase 
can  satisfy  the  LFT&E  requirement. 

c.  A  request  for  waiver  in  lieu  of  a  limited,  full-up 
sub-system/ system  LFT&E  program  can  also  be  perceived  by  system 
critics  as  a  cover-up  for  potential  system  deficiencies.  More 
importantly,  the  system  users  needs  to  have  as  complete  an 
understanding  as  possible  of  the  vulnerability/ lethality  strengths 
and  weaknesses  of  a  system  before  they  are  required  to  use  that 
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system  in  combat.  Thus,  the  question  should  not  be  "Can  we  afford 
to  conduct  LFT&E?",  but  "Can  we  afford  NOT  to  conduct  LFT&E?”  if  we 
are  to  ensure  that  "surprises”  are  learned  during  system 
development  and  not  on  the  battlefield. 


Section  III 

Independent  Evaluation  Plan/Test  Design  PLan  (lEP/TDP) 

3-7.  Defining  Critical  Issues 

The  lEP/TDP  defines  the  critical  issues  that  form  the  basis  for  the 
LFT&E  program  and  provides  the  crosswalk  between  the  critical 
issues  and  the  data  requirements.  Additionally,  the  data  sampling 
plan  and  analysis  techniques  are  specified  to  ensure  the  logic  of 
the  evaluation  is  understandable.  The  lEP/TDP  will  include  a 
section  describing  the  types  of  threats  or  targets  that  the  system 
is  expected  to  encounter  during  the  operational  life  of  the  system 
and  the  key  characteristics  of  the  threats/ targets  which  affect 
system  vulnerability/lethality.  A  reference  to  the  specific  threat 
definition  document/authority  will  be  presented  with  further 
discussion  of  the  rationale/criteria  used  to  select  the  specific 
threats/targets  or  surrogates  and  the  basis  used  to  determine  the 
number  of  threats /targets  to  be  tested  in  the  LFT.  Any  test 
limitations  or  shortfalls  and  their  impact  on  the  test  will  be 
identified.  Furthermore,  any  previous  data  that  will  be  used  to 
support  the  evaluation  will  be  discussed. 

3-8.  Independent  Developmental  Evaluator 

The  independent  developmental  evaluator  prepares  the  lEP/TDP  and 
addresses  all  aspects  of  the  evaluation  and  LFT  required  to  satisfy 
the  critical  issues.  The  lEP/TDP  is  the  responsibility  of  AMSAA 
with  assistance  provided  by  the  other  members  of  the  TIWG.  It  is  a 
stand-alone  document  and  must  be  developed  and  then  approved  by  the 
DUSA(OR)  six  months  prior  to  test  initiation.  The  approved  lEP/TDP 
will  also  be  submitted  to  the  DUSA(OR)  when  the  DTP  is  submitted 
for  approval. 


Section  IV 

Detailed  Test  Plan  (DTP) 

3-9.  Contents 

The  DTP  provides  explicit  instructions  for  the  conduct  of  the  LFT. 
It  is  prepared  by  the  technical  tester  and  is  derived  from  and 
implements  the  requirements  of  the  AMSAA  lEP/TDP.  The  exact 
format  can  vary  depending  on  the  test  program  but,  as  a  minimum,  it 
should  contain  individual  sections  which  address  the  major 
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categories  listed  below. 

a.  Introduction.  This  section  should  contain  a  summary 
description  of  the  test  program,  the  principal  participants  and 
their  roles,  the  test  item  and  its  perforaance  characteristics, 
previous  vulnerability  or  lethality  testing,  the  test  objectives, 
and  any  other  information  that  supports  LFT. 

b.  Test  Conduct.  This  section  covers  how  the  test  will  be 
conducted,  which  threats  or  targets  are  being  used,  what 
surrogates,  if  any,  will  be  used,  what  procedures  will  be  used  to 
ensure  test  discipline,  how  threats  will  be  fired/ launched,  what 
potential  lack  of  realism  may  result  from  absence  of  components, 
from  use  of  surrogate  components,  from  the  inerting  of  fuzes  on 
stowed  ammunition,  etc.  A  tabular  listing  of  all  threats/munitions 
to  be  fired  and  target  impact  conditions/ locations  will  be  provided 
via  summary  tables;  pictorial  representations  of  each  target  impact 
location  and  attack  angle  will  also  be  provided.  Finally,  the 
procedures  to  be  used  for  the  crew  casualty  and  system  damage 
assessments  will  be  described. 

c.  Appendices.  Individual  appendices  should  be  used  to 
address  subjects  such  as: 

(1)  System  Configuration.  This  appendix  describes  the 
target  configuration  and  its  fidelity  (i.e.,  BH&T;  full-up,  target 
simulants,  etc.)  and  discusses  how  the  test  item  compares  to  the 
actual,  combat  configured  target.  All  stowage  plans  for  full-up 
targets  will  be  pictorially  presented  to  show  locations  and 
quantity  of  items  stowed  on-board  (as  configured  for  combat) . 

These  stowage  plans  will  be  approved,  by  the  combat  user  for  U.S. 
systems  and  by  the  intelligence  community  for  foreign  systems, 
before  they  are  used  in  the  LFT.  A  more  detailed  discussion  of 
system  configuration  is  found  in  paragraph  5.2. 

(2)  Instrumentation  Plan.  This  appendix  describes  the 
instrumentation  suite  required  to  record  test  conditions  and 
measure  system  response  (e.g.,  projectile  striking  velocity,  fuel 
temperature,  component  acceleration,  etc.).  The  tester  will  define 
specific  instrumentation  requirements  based  on  the  lEP/TDP  data 
requirements . 

(3)  Battlefield  Damage  Assessment  and  Repair.  This 
appendix  defines  the  level  of  BDAR  to  be  performed  and  describes 
team(s)  membership,  repair  skill  level  requirements,  times  for 
repair,  etc.  The  BDAR  team(s)  support  required  will  be  decided  on 
a  case— by— case  basis  depending  on  the  fidelity  of  the  target. 
Typically,  BDAR  team(s)  perform  crew,  organizational  support, 
and/or  direct  support  levels  of  repairs. 
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(4)  The  OPSEC  Plan.  This  plan  is  included  as  part  of  the 
DTP  to  ensure  that  all  test  participants  are  aware  of  the  security 
aspects  of  the  LFT  and  how  the  data  are  to  be  handled. 

Furthermore,  the  high  visibility  and  sometimes  controversial  nature 
of  LFT  requires  strict  compliance  with  OPSEC  safeguards  and  a 
public  affairs  plan  to  cover  any  questions  asked  by  outside 
activities  or  private  citizens. 

3-10.  Preparation  and  Staffing 

The  DTP  is  prepared  by  the  tester  and  coordinated  with  members  of 
the  LFT&E  working  group.  After  coordination,  two  copies  of  the 
DTP,  along  with  two  copies  each  of  the  previously  approved 
J^SAA  lEP/TDP  and  the  SLAD/BVLD  Pre-Shot  Prediction  Report,  are 
forwarded  to  the  DUSA(OR)  at  least  60  days  prior  to  test 
initiation.  The  DTP  is  either  approved  by  the  DUSA(OR)  or  returned 
to  the  tester  for  changes/corrections. 


j  TESTING  WILL  NOT  START  UNTIL  THE  DTP  IS  APPROVED  BY  THE  DUSA(OR)  | 

I _ I 


3-11.  OSD  Review 

The  Army  approved  DTP,  along  with  the  Army  approved  lEP/TDP,  and 
the. Pre-Shot  Prediction  Report  are  forwarded  to  OSD  for  review  and 
comment;  OSD  suggested  changes  are  reviewed  by  the  DUSA(OR)  and 
incorporated  by  the  appropriate  lead  activity  as  directed  by  the 
DUSA(OR) .  The  DTP  must  also  outline  the  detailed  procedures  to  be 
followed  to  accommodate  unexpected  changes  to  the  LFT  that  may 
occur  during  actual  testing.  When  a  change  to  the  approved  DTP  is 
required,  it  is  essential  that  strict  adherence  to  the  change 
procedures  be  followed  to  avoid  repeating  test  shots  and  to  dispel 
any  perceptions  of  fixing  the  test  to  achieve  desired  results.  The 
TECOM  takes  the  lead  in  coordinating  changes  to  the  DTP  and  ensures 
these  changes  are  fully  coordinated  with  all  participating  LFT&E 
agencies.  Written  notification  of  the  proposed  change (s)  is 
forwarded  to  the  DUSA(OR)  for  approval.  No  change  from  the  DTP  is 
undertaken  until  approved  by  the  DUSA(OR)  and  provided  to  OSD  for 
review  and  comment.  After  approval,  all  participating  agencies  are 
notified  of  the  change  approval.  The  change  will  also  be 
documented  in  the  DTR  along  with  the  supporting  rationale. 


Section  V 

Pre-Shot  Prediction  Report 
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3-12.  General 

The  Pre-Shot  Prediction  Report  provides  the  vulnerability/ 
lethality  analysts'  best  estimate  of  the  expected  outcome  of  each 
shot  prior  to  the  actual  test  conduct  (i.e.,  a  pre-shot^ 
prediction) .  It  is  a  requirement  for  all  LFTs  and  provides  a 
snapshot  of  the  vulnerability/lethality  analysts'  current 
understanding  of  the  munition/target  interaction.  These 
predictions  can  range  from  subjective  engineering  judgments  of  the 
expected  damage  level  through  computer  generated  estimates  of  crew 
casualties  and  system  probabilities  of  kill  (P^s) .  The  SLAD/BVLD 
is  responsible  for  generating  the  pre-shot  predictions  for  each 
shotline.  Appropriate  pre-shot  prediction  techniques  will  be 
determined  by  SLAD/BVLD  on  a  case-by-case  basis  and  will  be 
consistent  with  the  technique  planned  for  casualty/damage 
assessment.  The  SLAD/BVLD  will  prepare  the  Pre-Shot  Prediction 
Report;  it  must  be  submitted  to  the  DUSA{OR) ,  along  with  the  DTP, 
60  days  prior  to  test  initiation.  The  Army  approved  Pre-Shot 
Prediction  Report  is  forwarded  along  with  the  DTP  and  the  lEP/TDP 
to  OSD  for  review  and  comment. 

3-13.  Predictions 

The  pre-shot  predictions  are  necessary  for  the  following  reasons: 


a.  To  ensure  useful  insights  will  be  gathered  about  the 
relative  vulnerability  or  lethality  of  the  system  involved. 

b.  To  establish  a  baseline  estimate  of  the  understanding  of 
the  munition/target  interaction  prior  to  test. 

c.  To  assist  in  shot  prioritization  from  least  to  most 
damaging.  This  will  ensure  that  most  of  the  testing  will  be 
completed  before  the  high  risk  shots  are  fired.  This  works  well 
for  both  vulnerability  and  lethality  tests  since  target  repair  is  a 
major  driver  in  the  turnaround  time  between  LFT  shots. 


Section  VI 

Detailed  Test  Report  (DTR) 

3-14.  General 

The  DTR  provides  a  formal  detailed  record  of  the  test  data  and 
information  obtained  during  the  conduct  of  the  LFT  and  describes 
the  conditions  which  actually  prevailed  during  test  execution  and 
data  collection.  The  test  report  documents  all  individual  shot 
test  conditions  and  test  results  required  by  and  identified  in  the 
DTP  and  approved  changes  to  the  DTP.  Sixty  days  after  test 
completion  the  DTR  is  provided  to  AMSAA,  to  support  their 
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independent  evaluation,  and  forwarded  to  the  DUSA(OR)  for  approval. 
The  approved  DTR  and  lER  must  be  forwarded  to  OSD  (DLFT)  within 
120  days  after  test  completion  and  45  days  prior  to  the  full-rate 
production  decision.  Schedules  must  be  planned  accordingly  to 
accommodate  these  mandatory  reporting  milestones. 


Section  VII 

Independent  Evaluation  Report  (lER) 

3-15.  General 

The  lER  documents  the  independent  evaluation  of  the  LET  and 
contains  the  assessment  of  the  critical  issues  and  conclusions 
concerning  the  vulnerability/ lethality  of  the  system.  The  lER  is 
the  sole  responsibility  of  AMSAA,  the  technical  independent 
evaluator.  The  lER  addresses  the  test  objectives,  issues,  and 
criteria  as  defined  in  the  lEP/TDP.  It  discusses  the  crosswalk 
between  results  and  the  evaluation  and  specifies  any  limitations 
relative  to  the  analysis.  All  aspects  of  the  test  will  be 
evaluated,  both  negative  and  positive.  The  evaluation  will  be 
balanced  by  the  discussion  of  vulnerability/ lethality  based  on  the 
likelihood  of  occurrence  on  the  battlefield.  The  lER  is  submitted 
to  the  DUSA(OR)  for  review  and,  together  with  the  DTR,  is  forwarded 
to  OSD  within  120  days  after  test  completion.  The  lER  and  all 
LFT&E  reports  (to  include  the  OSD  report  to  Congress)  must  be 
rendered  prior  to  the  Milestone  III  full-rate  production  decision. 


Section  VIII 

Detailed  Daunage  Assessment  Report 
3-16.  General 

This  report  documents  the  detailed  analyses  and  crew  casualty  and 
system  damage  assessments  of  the  individual  test  events.  It 
includes  an  in-depth  comparison  of  the  pre-shot  predictions  of  crew 
and  system  damage  and  the  observed  test  outcomes.  This  process 
requires  a  detailed  examination  of  component  damage  states,  failure 
modes,  damage  mechanisms,  etc.,  to  ensure  a  full  understanding  of 
model  predictive  capability.  Anomalies  will  be  identified  and,  if 
required,  model  updates  specified.  These  in-depth  analyses  will 
not  preclude  SLAD/BVLD  from  providing  its  required  support  to  the 
LFT  evaluation.  The  individual  shot  damage  assessment  records  will 
be  provided  by  SLAD/BVLD  to  the  tester  within  30  days  after  test 
completion  and  subsequently  to  AMSAA  to  support  its  independent 
evaluation.  Within  six  months  after  completion  of  the  test,  the 
SLAD/BVLD  will  publish  the  Detailed  Damage  Assessment  Report. 
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Chapter  4 
Modeling 

Section  I 
Introduction 


a.  Much  of  the  early  controversy  surrounding  LFT&E  focused  on  the 
adequacy  of  existing  Army  vulnerability/lethality  models  and  their 
appropriate  role  in  the  overall  LFT&E  process.  Too  often  people 
interpreted  the  debate  over  these  issues  in  such  a  manner  that 
modeling  and  testing  were  viewed  as  an  either-or  proposition.  The 
basic  fact  is  that  both  are  needed  and  are  essential  to  a 
comprehensive  and  effective  LFT&E  program.  They  are  complementary 
efforts  and  the  LFT&E  strategy  and  planning  must  be  based  on  this 
view.  This  section  will  attempt  to  provide  a  better  understanding 
of  the  Army's  vulnerability/ lethality  models  and  their  role  in 
LFT&E. 


b.  Live  Fire  Testing,  even  when  supplemented  with 
developmental  testing,  cannot  produce  enough  data  to  assess  the 
vulnerability  or  lethality  of  a  system  for  all  combinations  of 
threat,  impact,  and  engagement  conditions.  Thus,  modeling  must  e 
used  to  extend  test  results  to  account  for  conditions  impractical 
or  impossible  to  test.  The  reader  is  reminded  that  modeling  here 
is  defined  in  the  broad  sense  given  in  Chapter  1. 


Section  II 
Role  of  Modeling 

In^the^context  of  LFT&E,  vulnerability/ lethality  modeling  has  four 
basic  roles:  support  test  design,  support  the  evaluation  of  system 
and  crew  vulnerability  or  munition  lethality,  guide  and  evaluate 
vulnerability  reduction  or  lethality  enhancement  efforts,  and 
methodology  diagnosis. 

4-3.  Test  Design  Support  ,  u..  •.  4. 

Live  Fire  Testing  is  expensive  and  it  is  absolutely  essential  that 
the  maximum  information  be  collected  with  the  resources  allocated 
to  LFT&E.  Modeling  is  used: 

a.  To  determine  which  engineering  shots  make  the  most  sense  in 
terms  of  what  is  known  about  the  vulnerability  or  lethality  of  the 
system  being  tested,  the  expected  performance  of  the  threat 
munitions  or  target,  and  the  specific  evaluation  issues  for  the 
system  being  tested; 
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b.  To  develop  and  apply  exclusion  rules  for  randomly  selected 
shots  and,  once  those  shots  have  been  selected,  to  determine  from 
pre-shot  predictions  which,  if  any,  should  be  conceded  to  avoid 
unnecessary  loss  of  test  assets; 

c.  To  "filter”  random  and/or  engineering  shot lines  to  ensure  a 
specified  level  of  damage  will  be  considered  (e.g. ,  using  LOF 
matrices  to  identify  weapon/target  impact  locations  which  satisfy  a 
preselected  criteria  that  only  "shotlines  with  a  LOF  greater  than 
or  less  than  a  certain  value  will  be  considered". 

4-4.  Evaluation  Support 

Model  outputs,  in  conjunction  with  Live  Fire  and  development  test 
results,  are  used  by  AMSAA  to  address  critical  evaluation  issues 
pertaining  to  system  vulnerability  or  lethality,  crew  casualties, 
and  logistic  supportability.  It  is  difficult  to  separate 
vulnerability  or  lethality  evaluations  directly  supporting  LFT  from 
those  required  to  support  the  entire  acquisition  process  because, 
in  a  broader  context,  model  generated  vulnerability  and  lethality 
estimates  are  critical  inputs  to  system  effectiveness  studies,  such 
as  the  cost  and  operational  effectiveness  analysis  (COEA) ,  designed 
to  determine  force  exchange  ratios,  optimum  tactical  deployment 
schemes,  wartime  maintenance  and  medical  requirements,  and  other 
measures  of  system  cost  and  benefit. 

4-5.  Vulnerability  Reduction/Lethality  Enhancement 
Modeling  also  supports  vulnerability  reduction  and  lethality 
enhancement  efforts  by  allowing  the  analyst  to  evaluate  the 
potential  payoff  of  design  changes  intended  to  reduce 
casualties/system  vulnerability  or  increase  munition  lethality. 

4-6.  Methodology  Diagnosis 

One  objective  of  LFT  is  to  determine  the  extent  to  which  the 
vulnerability  and  lethality  models  account  for  all  pertinent 
munition  damage  mechanisms  and  target  failure  modes.  In  this 
context,  modeling,  via  comparing  pre-shot  predictions  with  test 
results,  can  provide  insights  into  the  fidelity  of  the  models 
themselves.  Seldom  will  enough  data  be  generated  from  a  single  LFT 
to  allow  a  complete  verification  of  model  performance.  But, 
insights  can  be  gained  to  suggest  whether  significant 
munition/target  interactions  are  being  neglected  by  the  models  and 
to  identify  areas  of  model  performance  which  need  to  be  more 
thoroughly  examined  in  on-going  model  improvement  programs. 


Section  III 
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Assessment  Techniques 
4-7.  Techniques 

Vulnerability  or  lethality  modeling  can  be  as  simple  as  using  a 
series  of  charts  to  determine  whether  missile  fragments  are  likely 
to  sever  a  drive  shaft  in  the  LFT,  or  a  subset  of  the  LFT, 
conducted  on  a  component  or  sub— system  level.  At  the  other 
extreme,  modeling  may  involve  the  use  of  several  large  scale 
computer  codes  to  generate  distributions  of  system  probabilities  of 
kill  given  a  hit  (Pkfl,s)  which  take  into  account  all  known 
munition/ tar get  interaction  phenomena  and,  in  addition,  address  the 
stochastic  nature  of  these  interactions.  In  general,  more  than  one 
model  must  be  used  to  characterize  such  phenomena  as  target 
geometry,  munition  performance,  armor  performance,  BAD,  personnel 
injuries,  component  and  sub— system  failure  modes,  aircraft  airspeed 
and  altitude  dependence,  and  component  kill  probabilities. 

Usually,  these  models  are  implemented  and  applied  with  personal  and 
mainframe  computer  codes  which,  depending  on  their  complexity  and 
sophistication,  have  modules  to  implement  these  models  or  use  as 
input  the  products  of  auxiliary  codes.  It  is  important  to 
recognize  that  the  choice  of  models  cannot  be  specified 
arbitrarily.  Rather,  the  appropriate  model  or  assessment  technique 
must  be  chosen  on  the  basis  of  how  much  is  known  about  the  threat 
munition  or  target,  input  data  that  are  available,  and  perhaps  most 
importantly,  the  vulnerability  or  lethality  issues  that  the  LFT  is 
designed  to  address.  While  the  most  detailed  and  sophisticated 
models  consistent  with  these  criteria  should  always  be  used,  it  is 
not  unusual  for  one  suite  of  models  to  be  most  appropriate  for 
pre-shot  predictions  while  another  suite  of  models  is  best  for  some 
other  aspect  of  the  LFT&E  effort.  This  flexibility  in  model 
selection  is  especially  necessary  for  lethality  LFT&E  because  the 
level  of  knowledge  of  the  threat  target  is  often  extremely  limited. 

any  given  LFT,  be  it  vulnerability  or  lethality,  the  suite  of 
analysis  models  must  be  selected  by  the  vulnerability/ lethality 
analyst  in  coordination  with  AMSAA.  However,  once  this  choice  of 
assessment  technique  is  made,  it  is  important  to  create  an  audit 
trail.  The  underlying  rationale  for  the  model  or  its  modification, 
model  limitations,  assessment  procedures,  and  required  input  data 
should  be  documented.  The  models  to  be  used  must,  of  course,  be 
specified  in  the  lEP/TDP.  However,  depending  on  the  level  of 
development  of  the  LFT&E  strategy,  they  may  or  may  not  be 
identified  in  the  earliest  versions  of  the  TEMP. 


Section  IV 
Data  Bases 

4-8.  General 
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Regardless  of  the  specific  models  selected  to  support  any  given  LFT 
there  are  several  data  bases  that  must  be  developed  prior  to  LFT. 
The  exact  nature  of  these  data  bases  will,  of  course,  vary 
depending  on  the  models  actually  used.  However,  they  will  usually 
include  such  things  as  component  Pk/i,s,  target  descriptions,  threat 
munition  and  armor  performance,  BAD  characteristics,  failure  modes 
and  component/ sub-system  criticality,  kill  criteria,  DALs, 
helicopter  altitude-airspeed  diagrams,  and  the  sensitivity  of 
combustibles  to  fragment  and  penetrator  impacts.  Development  of 
these  supporting  data  bases  must  begin  one  to  two  years  in  advance 
of  the  start  of  the  LFT.  A  potential  problem  with  the  scheduling 
of  tests  and  analyses  to  generate  these  data  bases  is  that  the  data 
must  be  pertinent  to  the  planned  production  design  of  the  system  or 
munition  being  tested.  For  example,  penetration  characteristics 
for  a  new  projectile  must  be  for  the  production  design  as  opposed 
to  evolutionary  development  prototypes.  Some  of  these  data  bases 
will  be  developed  wholly  or  in  part  to  support  the  overall  T&E 
process;  others  are  needed  to  directly  support  LFT.  In  any  event, 
costs  and ^hardware  reguirements  must  be  identified  as  early  as 
possible  in  the  TEMP  in  order  to  permit  their  inclusion  in  budget 
and  contractual  documents. 


Section  V 

Vulnerability/Lethality  Estimates 
4-9.  General 

Vulnerability  and  lethality  estimates  in  the  form  of  Pk/hS, 
vulnerable  areas,  and  blast  contours  are  typically  generated  by,  or 
under  the  auspices  of,  the  SLAD/BVLD.  (For  JLF  Programs  and  Army 
LFT  of  multiservice  equipment  or  munitions,  vulnerability/ lethality 
modeling  may  be  conducted  or  supported  by  the  Navy  or  Air  Force.) 
These  vulnerability  and  lethality  estimates  are  essential  inputs  to 
system  effectiveness  studies;  they  also  provide  a  basis  for 
relative  comparisons  (e.g.,  to  determine  whether  the  requirement  to 
reduce  the  average  P^^^  or  vulnerable  area  by  some  amount  has  been 
met) .  However ,  the  vulnerability  and  lethality  estimates  do  not 
account  for  combat  attack  distributions,  deployment  conditions,  or 
weapon  hit  probabilities.  Typically,  AMSAA  applies  these  factors 
to  the  vulnerability  and  lethality  estimates  to  generate  P^s  given 
a  shot,  burst,  etc.  These  P^s  are  then  used  by  AMSAA,  TRADOC,  or 
other  agencies  to  evaluate  system  survivability  or  firepower  to 
determine  force  exchange  ratios,  identify  maintenance  requirements, 
or  determine  some  other  measure  of  system  effectiveness.  Thus, 
there  is  clearly  a  critical  link  between  vulnerability/ lethality 
modeling  and  system  level  evaluations.  It  is  evident  that 
vulnerability  and  lethality  analyses  must  be  responsive  to  the 
requirements  of  the  system  level  studies.  Conversely,  evaluation 
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Faks-^:  Ks.ia™s 

ifinoS'inpu?  dItS  blses  to  be  developed  and  necessary  nodel 
modifications  to  be  made. 


Section  VI 

Classes  of  Models /Algorithms 

nlre  arrfgreat  number  of  models  or  algorithms  used  to  support 

the  vulnerability/ lethality  assessment  process.  In 

Jahio  i-i  three  classes  of  such  models  are  compared  for  output 

factors  associated  with  vulnerability/ lethality  models. 
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Table  4-1.  Comparison  of  Three  Types  of  Vulnerability  Models 


Model  Tvoe 

Output 

Measures 

Level  of  Detail 

Aoolications 

Lumped  Parameter 
(e.g..  Compartment) 

Expected  M-Kill 
Expected  F-Kill 
Expected  M/F-Kill 
Expected  K-Kill 

Structure 

External  suspension 
Compartments  ( crew , 
ammo,  engine) 

Crew  casualty 

COEAs,  MAAs, 
SSEBs , 

Compartment- 
level  trade¬ 
off  analyses. 
Vulnerability 
reduction 

Expected  Value 

Point  Burst 

(e.g.,  VAST,  HEVART) 

Same  as  above  plus 
component  P^s 
Attrition 

Forced  Landing 
Mission  Abort 

Repair  Times 

Structure 

Suspension 

Components 

Crew  casualty 

Same  as  above 
plus  com¬ 
ponent  level 
trade-off 
analyses, 
SPARC 
analyses 

Stochastic  Point 

Burst 

(e.g.,  SPRAE,  SQuASH) 

M-kill  Pdf 

F-Kill  Pdf 

M/F-Kill  Pdf 

K-Kill  Pdf 

Component  P^s 
Component  Damage 
State  Pdf 

Same  as  above 

Same  as  above 
plus 

estimation  of 
errors  in 
field 
sampling, 
propagation 
of  uncertain- 
ities,  and 
calibration 
of  lower- 
level  models. 

Legend : 

F-kill  =  Firepower  kill 

K-kill  =  Catastrophic  kill 

M-kill  =  Mobility  kill 

M/F-kill  =  Mobility  or  Firepower  kill 


HEVART 

MAA 

Pdf 

SSEB 


High  Explosive  Vulnerable  Areas  and  Repair  Times 
Mission  Area  Analysis 
Probability  density  function 
Source  Selection  Evaluation  Board 
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SPRAE  =  Stochastic  PRocessor  of  Artillery  Effects 

SQUASH  =  Stochastic  Qualitative  Assessment  of  System  Hierarchies 

VAST  =  Vulnerability  Analysis  of  Surface  Targets 
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Chapter  5 

Test  Conduct  and  Parameters 


Section  I 
Introduction 

5-1.  General 

This  section  provides  general  guidance  for  the  conduct  of  LFT  and 
discusses  those  parameters  and  functions  which  must  be  considered 
during  test  planning  (e.g.,  vehicle  stowage,  instrumentation, 
scheduling,  etc.);  actual  test  requirements  will  be  established  on 
a  case-by-case  basis  to  address  the  data  requirements  defined  in 
the  lEP/TDP.  Guidance  presented  in  this  section  is  based  on  Army 
LFT&E  experience  to  date  with  armor/anti-armor  systems;  test 
conduct,  test  parameters/ functions,  and  terminology  discussed  in 
the  following  paragraphs  reflect  this  experience.  Because  the 
primary  purpose  of  LFT&E  is  to  address  crew  survivability,  most  of 
the  parameters/ functions  and  the  testing  discussed  in  this  section 
is  applicable  to  any  type  of  system  and  the  remaining  items  are 
easily  extrapolated  to  other  types  of  systems.  Again,  the  reader 
is  cautioned  that,  all  requirements  must  be  determined  on  a  case- 
by-case  basis. 


Section  II 

Test  Item  Configuration 
5-2.  Vulnerability  LFT&E 

a.  Vulnerability  LFT&E  is  conducted  to  identify  potential 
system  integration  vulnerabilities  which  cannot  be  adequately 
addressed  through  component  and/or  sub-system  testing.  In  order  to 
provide  the  most  realistic  test  possible  and  to  accurately  assess 
the  vulnerability  of  the  system  and  the  survivability  of  the  crew, 
the  weapon  system  must  be  as  close  to  its  combat  configuration  as 
possible.  Combat  configuration  denotes  a  fully  operational  test 
item  complete  with  all  sub-systems  and  on-board  stowage  items. 

b.  The  presence  of  a  fully  operational  test  item  with  all 
sub-systems  is  particularly  important  in  evaluating  ballistic  shock 
damage  and  the  interaction  between  sub-systems  as  a  result  of 
damage  to  different  components.  In  order  that  the  individual 
effects  of  each  shot  on  the  test  item  can  be  determined,  the  test 
item  is  repaired  and  baseline  performance  characteristics 
determined  prior  to  each  test  shot.  Baseline  procedures  should 
include  a  complete  functional  check  of  all  major  sub-systems  on  the 
test  item  and  may  also  include  performance  checks  such  as  mobility 
or  firepower  characteristics. 
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c.  Systems  undergoing  LFT&E  testing  are  stowed  in  a  combat 
configuration  so  that  the  effects  of  the  stowage  on  the  system 
vulnerability  and  crew  survivability  can  be  assessed.  Stowage  in  a 
combat  configuration  includes  ammunition,  fuel,  additional 
authorized  list  (AAL)  items,  and  basic  issue  items  (BII) . 
Anthropomorphic  simulants  and/or  wooden  mannequins  are  located  in 
crew  positions  as  an  aid  in  crew  survivability  assessments. 
Ammunition  should  be  live,  with  inert  fuzes  or  fuzes  removed  (live 
fuzes  damaged  during  test  conduct  could  present  a  hazard  to  test 
personnel) .  However,  if  the  reaction  correlation  between  inert  and 
live  ammunition  is  known  and  predicable,  inert  rounds  may  be  stowed 
to  ensure  survivability  of  limited  assets.  The  use  of  inert  rounds 
in  lieu  of  live  ammunition  will  be  approved  by  the  DUSA(OR)  on  a 
case-by-case  basis.  Any  planned  shot  which  the  PM  considers  to  be 
catastrophic  or  of  significant  damage  may  be  conceded;  however, 
conceded  shots  will  be  assigned  a  P^  =  1.0  for  the  evaluation. 

d.  All  fuel  in  the  test  item  will  be  at  normal  operating 
temperatures  for  the  system  at  the  time  of  the  test  firing.  This 
is  necessary  since  the  flammability  of  the  fuel  increases  as  its 
temperature  increases.  Typically,  this  is  accomplished  by  adding 
heated  fuel  to  the  test  item  prior  to  the  test  firing. 

e.  The  AAL  and  BII  are  stowed  on  the  test  item  in  accordance 
vith  an  approved  stowage  plan.  Typically  the  stowage  plan  is 
developed  by  the  responsible  TRADOC  school  and  verified  by  the 
tester  prior  to  testing.  Crew  simulants  are  dressed  in  the 
appropriate  ensemble  to  include  helmet,  personal  weapons,  goggles, 
gloves,  boots,  coveralls,  ballistic  vest,  and  battle  dress  uniform, 
as  prescribed  by  Army  doctrine.  This  assures  that  the 
anthropomorphic  simulant  or  wooden  mannequin  is  representative  of 
an  actual  crew  member  and  that  the  protective  features  of  the 
uniform  are  accounted  for  in  the  crew  injury  evaluation. 

f.  A  hazard  analysis  is  performed  on  all  of  the  stowage 
items.  Any  stowage  item  which  could  pose  a  hazard  to  test 
personnel,  if  damaged  during  testing,  must  be  modified  or  replaced. 
Those  items  modified  or  replaced  must  be  listed  in  the  DTP.  For 
example,  certain  types  of  chemical  detectors  used  on  combat 
vehicles  contain  a  radioactive  isotope  as  part  of  the  sensor— this 
isotope  would  be  removed  prior  to  stowing  the  detector. 


5-3.  Lethality  LFT&E 

a.  Lethality  LFT&E  is  conducted  to  demonstrate  the 
effectiveness  of  U.S.  munitions  against  required  or  representative 
threat  targets.  Targets  for  lethality  LFT&E  can  include  target 
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simulants  (i.e.,  targets  constructed  to  represent  a  shotline 
against  a  known  threat) ,  BAD  collection  packets  to  determine 
residual  penetration  and  spall  data,  and,  if  possible,  full-up 
systems.  The  full-up  system  could  be  the  actual  threat,  if 
available,  an  available  "older”  threat,  or  an  approved  surrogate 
(see  Chapter  2  and  7) .  The  actual  targets  and  full-up  systems  to 
be  used  are  determined  on  a  case-by-case  basis  and  will  be 
specified  in  the  LFT&E  portion  of  the  TEMP. 

b.  As  with  vulnerability  LFT&E,  the  full-up  threat  system 
will  be  in  a  full  combat  configuration  (i.e.,  fully  operational  and 
stowed  per  an  approved  stowage  plan) .  The  full-up  system  test 
provides  a  mechanism  for  evaluating  overall  munition  effectiveness 
due  to  penetration/perforation,  spall,  ballistic  shock,  fire,  blast 
overpressure,  toxic  fumes,  etc.  The  use  of  inert  ammunition  in 
lethality  LFT&E  is  subject  to  the  same  conditions  given  in  the 
preceding  paragraph. 

5-4 .  Simulated  Targets 

In  order  to  preserve  valuable  threat  assets  or  when  threats  are  not 
available  for  lethality  LFTs,  targets  constructed  to  represent  a 
given  threat  characteristic  can  be  used  in  lieu  of  full-up  targets. 
Tests  conducted  with  these  targets  should  be  used  to  supplement  a 
limited  full-up  LFT;  simulated  target  tests  alone  do  not  provide  an 
adequate  demonstration  of  a  system's  lethality.  Data  which  can  be 
obtained  from  simulated  target  testing  (either  directly  or  from 
modeling  efforts)  are:  profile  hole  diameters;  BAD  (fragment  mass, 
velocity,  and  spatial  distribution) ;  residual  penetration;  and 
individual  P^s  for  a  selected  target  impact  location.  If  side-by- 
side  testing  of  two  or  more  munitions  is  conducted,  statistical 
tests  (e.g..  Sign  Test,  student  t-test,  etc.)  can  be  used  to 
conduct  lethality  comparisons. 


Section  III 

Test  Assets  and  Schedule 
5-5.  General 

The  LFT  is  normally  the  last  test  to  be  conducted  prior  to  the 
full-rate  production  decision  and,  as  such,  planning  and  resourcing 
must  be  addressed  early-on  in  the  LFT&E  program.  The  strategy  and 
resource  requirements  (to  include  targets/munitions)  to  accomplish 
an  efficient  and  effective  LFT&E  program  must  be  included  in  the 
TEMP  T&E  Resource  Section. 

5-6 .  Conduct 

Conduct  of  LFT  is  driven  by  the  time  between  shots  required  to 
repair  the  target.  Full-up  system  tests  usually  require  extensive 
repairs  and  repair  time.  Experience  indicates  that  there  is  a 
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three-to-one  ratio  of  repair  time  to  test  range  time.  To  increase  j 

test  efficiency  and  provide  maximum  utilization  of  personnel  and 
hardware,  it  is  advantageous  to  conduct  LFTs  with  multiple  target 
assets.  Multiple  target  assets  allow  for  overlapping  of  test  and 
repair  time,  thus,  increasing  testing  efficiency.  When  multiple 
test  assets  are  not  feasible,  the  LFT&E  schedule  must  reflect  the 
total  time  required  to  complete  the  testing.  If  the  schedule 
cannot  accommodate  these  time  requirements,  it  may  be  necessary  to 
restructure  the  strategy.  Decisions  concerning  assets,  schedules, 
and  strategy  are  addressed  by  the  LFT&E  working  group.  As  with 
other  phases  of  the  T&E  process,  unresolved  issues  are  forwarded  to 
higher  headquarters  for  resolution. 

Section  IV 
Instrumentation 

5-7.  Data  Collection 

Test  assets  and  LFT  are  expensive;  therefore,  a  complete  set  of 
data  must  be  gathered  on  each  shot  to  facilitate  the  crew  and 
system  damage  assessment,  to  measure  and/or  record  test  conditions, 
and  to  ensure  test  conformity  (i.e.,  compliance  with  the  DTP).  In 
addition  to  instrumentation  for  addressing  crew/ system  damage,  the 
test  item  is  instrumented  to  provide  early  warning  of  potential 
problems  resulting  from  the  test  event.  Parameters  measured  could 
include:  engine  rpm,  voltage,  hydraulic  fluid  pressures  and 

temperatures,  oil  pressures  and  temperatures,  coolant  temperatures, 
and  automatic  fire  suppression/fire  extinguishing  system 
discharges.  Actual  instrumentation  suites  are  determined  by  the 
tester  on  a  case-by-case  basis  to  address  the  lEP/TDP  data 
requirements  and  test  item  safety/ security  requirements.  These 
instrumentation  packages  typically  include  the  following; 

a.  Video  and  high  speed  photography  to  provide  visual 
documentation  of  the  test  event.  Video  documentation  provides  real 
time  monitoring  of  the  interior  and  exterior  of  the  test  item.  The 
exterior  video  also  provides  assistance  in  locating  parts  displaced 
by  the  munition/target  interaction.  The  internal  video  provides 
real-time  infoirmation  on  perforation  of  the  target  protective 
system,  the  presence  and  extent  of  internal  fires,  and  test  item 
status  information  required  for  determining  when  it  is  safe  for 
test  personnel  to  re-enter  the  test  site. 

b.  Projectile  flight/performance  instrumentation  to  record 
striking  velocity,  velocity  profile  from  launch  to  impact, 
pitch/yaw  history,  and  penetration  characteristics.  Video  cameras, 
high  speed  cameras,  and/or  flash  x-rays  may  be  used. 

c.  Toxic  fumes  instrumentation  to  record  the  levels  of 
potentially  hazardous  gases  (e.g.,  nitrous  oxide,  nitrogen  dioxide. 
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i  carbon  monoxide,  carbon  dioxide,  hydrogen  fluoride,  hydrogen 

bromide,  cyanide,  aldehydes,  etc.)  and  airborne  particulates. 

Toxic  fumes  data  are  collected  at  crew  member  locations.  Specific 
items  and  crew  locations  to  be  sampled  are  system  dependent  and 
will  be  determined  based  on  an  analysis  of  the  potential  hazard 
posed  by  on-board  materials. 

d.  Thermal  effects  instrumentation  to  record  temperature  and 
heat  flux  data  related  to  the  crew  and  test  item.  These  data  are 
used  to  assess  crew  survivability,  provide  engineering  data  to 
assess  hardware  vulnerability,  and  ensure  compliance  with  the  DTP 
parameters  (e.g.,  fuel  temperature  at  shot  time). 

e.  Blast  overpressure  instrumentation  to  record  pressure  time 
histories.  Overpressure  data  are  collected  in  the  crew  compartment 
and  external  to  the  test  item  to  assist  in  assessing  personnel 
casualties  and  to  provide  engineering  data  to  assess  hardware 
vulnerability. 

f.  Ballistic  shock  instrumentation  to  record  accelerations 
and  forces  on  the  crew  and  critical  system  components. 
Accelerometers,  strain  gages,  and/or  velocity  gages  can  be  placed 
on  components  to  measure  the  ballistic  shock  transmitted  through 
the  structure  of  the  test  item  to  the  components  and  on 
anthropomorphic  simulants  to  measure  acceleration  and  forces 
transmitted  to  the  crew.  The  simulants  are  positioned  in  crew 

/  locations  away  from  the  primary  penetrator  path/ spa 11  cone  where 

i  ballistic  shock  to  crew  is  of  concern.  Wooden  mannequins  can  be 

placed  in  other  crew  locations  to  record  the  effect  of  the 
penetrator /spa 11  cone. 

g.  Plate  arrays  and  BAD  packets  to  record  penetration 
performance,  residual  penetration,  and  spall  cone  characteristics 
(i.e.,  fragment  number,  size,  velocity,  and  spatial  distribution). 


Section  V 
Facilities 

5-8.  Facilitity  Considerations 

Live  Fire  Testing  often  requires  extensive  test  facility 
capabilities  to  allow  for  realistic  and  cost  effective  testing; 
actual  facilities  for  a  given  program  will  be  driven  by  the  test 
and  data  requirements.  Test  facility  capabilities  which  could  be 
required  to  support  a  given  program  are: 

a.  Multimunition  Firing.  The  threat  could  consist  of  gun 
fired  rounds,  missiles,  rockets,  mines,  etc.,  requiring  a  variety 
of. launching  capabilities.  Threats  could  require  real  range 
firings,  reduced  range  firings,  and  static  firings  (e.g.,  mine 

r 
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firing  in  prepared  soil  with  specified  density  and  moisture 
content) .  Launch  conditions  could  be  direct  fire,  superelevation 
(anti-air  simulation) ,  and  high  angle  of  fall  (indirect  fire 
simulation) . 

b.  Instrumentation  Suite.  Live  Fire  Testing  is 
instrumentation  intensive  and  could  require  upwards  of  200  channels 
of  data  collection  during  any  given  shot.  Substantial  video  and 
high  speed  film  coverage  for  documentation  and  test  item  security 
could  be  required. 

c.  Range/Test  Item  Security.  In  addition  to  video  to  provide 
real-time  visual  security,  an  auxiliary  fire  suppression  system 
could  be  required  for  protection  of  range  and  instrumentation  suite 
facilities  as  well  as  test  item  security.  Providing  adequate 
protection  to  instrumentation  cables  from  fragments  and/or  fire  to 
ensure  test  requirements  are  not  compromised  must  be  a  prime 
consideration.  Additionally,  environmental  protection  in 
accordance  with  federal/ state  government  mandates  must  be 
adequately  addressed  (i.e..  Environmental  Impact  Statements  must  be 
developed,  staffed,  and  approved  prior  to  test  initiation) . 


d.  Repair  Facility.  Because  test  assets  are  limited  and 
LFT&E  test  item/target  configuration  requirements  are  stringent, 
the  ability  to  perform  repairs  will  be  necessary.  These  repairs 
could  include  welding,  machining,  fabricating/replacing  damaged 
components,  and  major  reconstruction  of  the  test  item.  Repair  up 
to  Depot  Level  could  be  required. 


Section  VI 
Test  Discipline 


5-9.  General  ^  ^  o-  •  i- 

The  high-visibility  and  oversight  of  LFT  requires  strict  discipline 
during  the  conduct  of  the  testing.  This  paragraph  summarizes  key 
test  discipline  items  which  are  applicable  to  future  LFTs. 


a.  Follow  the  DTP.  One  of  the  primary  responsibilities  of 
the  tester  is  to  ensure  that  the  test  is  conducted  in  accordance 
with  the  HQDA  approved  DTP.  Unauthorized  deviations  from  the  DTP 
are  not  permitted.  Additionally,  THE  LPT  WILL  NOT  START  UNTIL  THE 
DTP  IS  APPROVED.  With  LFT&E  scheduled  near  the  critical  full-rate 
pj-Qduction  decision  milestone  and  test  shots  relatively  expensive, 
it  is  essential  that  the  DTP  be  followed  to  avoid  potential 
problems.  Conducting  the  test  according  to  an  approved  DTP  will 
eliminate  the  perception  of  bias  or  of  rigging  the  test  in  order  to 
ensure  positive  results.  Changing  shotlines,  threats,  stowage. 
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etc.,  even  for  sound  technical  reasons,  without  proper  coordination 
and  authorization  is  not  permitted. 

b.  Change  Procedures.  It  is  rare  that  a  LFT  is  conducted 
without  some  deviation  from  the  approved  DTP  being  required.  In 
order  to  address  these  potential  deviations  and  retain  testing 
integrity,  a  strict  procedure  has  been  adopted  for  approving 
changes  to  the  DTP.  This  change  procedure  is  described  in  Chapter 
3. 


c.  Reporting  Emerging  Results.  The  dissemination  of  emerging 
results  provides  test  participants  a  continuing  awareness  of  test 
progress  and  an  early  identification  of  potential 

vulnerability/ lethality  shortcomings.  Data  is  usually  disseminated 
at  data  review  meetings.  These  meetings  should  be  held 
periodically  throughout  the  test  so  that  data  can  be  reviewed, 
commented  on,  and  necessary  subjective  judgments  reviewed  for 
consistency  and  soundness.  Representatives  of  the  damage 
assessment  team  (DAT)  (see  paragraph  5-10) ,  PM,  and  system 
contractor  are  typically  present  at  these  meetings.  However,  it 
should  be  noted  that  for  purposes  of  assessing  the  shots,  the  PM 
and  system  contractor  have  no  vote,  but  are  present  to  provide 
information  on  system  design  characteristics,  if  required.  The  OSD 
will  have  access  to  these  meetings;  however,  any  results  addressed 
during  these  meetings  and  used  in  the  OSD  assessment  will  be 
provided  to  the  Army  for  factual  review  prior  to  its  use. 

Section  VII 

Damage/Casualty  Assessment 
5-10.  General 

After  each  shot  the  target  is  examined  and  the  system  damage  and 
crew  casualties  are  assessed.  This  paragraph  defines  the  Army 
approach  to  this  process. 

5-11.  Damage  Assessment  Team  (DAT) 

The  DAT  is  responsible  for  collecting  and  assessing  crew 
incapacitation  and/or  test  item/target  damage  after  each  shot.  The 
DAT  will  be  chaired  by  the  SLAD/BVLD  and  will  include  the  tester 
and  the  user  as  a  minimum.  Other  interested  organizations  will  be 
requested  to  support  the  DAT  as  required.  The  specific  tasks  of 
the  DAT  are  to: 

a.  Document  any  physical  damage  to  the  simulated  crew  member 
and  assess  the  extent  of  their  injuries  (i.e.,  level  of 
incapacitation) . 

b.  Document  any  physical  damage  to  the  test /target  item. 
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c.  Determine  if  any  injury,  degradation,  and/or  loss  of 
function  (LOF)  occurred  which  would  affect  the  ability  of  the  crew 
and  system  to  perform  their  mission. 

d.  Determine  the  damage  mechanisms  causing  any  injury, 
degradation,  and/or  LOF. 

e.  Characterize  the  test  item's  performance  and  other 
parameters,  before  and  after  each  shot,  to  allow  for  future 
vulnerability  reduction/ lethality  enhancements. 

f.  Document  and  characterize  behind-armor  effects  produced  by 
the  test  munition. 

g.  Utilize  the  preceding  information  to  assess  crew 
casualties  and  system  P^/h  values  for  the  test  munition. 

h.  Provide  a  final  damage  assessment  report  for  each  shot; 
necessary  subjective  judgments  will  be  based  upon  the  majority 
viewpoint  of  the  DAT. 

5-12.  crew  Vulnerability  .  -  ^ 

Crew  vulnerability  can  be  assessed  through  the  examination  of  data 
collected  with  crew  simulants  and  crew  environment  instrumentation. 

a.  Crew  simulants  can  be  used  to  assess  the  expected  damage 
to  the  crew  members.  The  following  simulants  have  been  used  in 
previous  LFTs : 


(1)  Fully  combat  dressed  wooden  manneguins  placed  in  crew 
positions  in  the  expected  penetrator  path/ spall  cone  where 
acceleration  injury  is  not  a  main  concern.  After  each  shot,  the 
fully  combat  dressed  mannequins  are  assessed  for  damage  (e.g., 
burns  on  clothing,  damaged  body  parts,  fragment 
penetration/perforation,  etc.). 

(2)  Fully  combat  dressed  anthropomorphic  simulants 
("anthros")  placed  in  crew  positions  where  acceleration  is  the  main 
concern.  "Anthros"  can  be  used  to  measure  triaxial  acceleration, 
compression,  biaxial  bending,  fore-aft  bending,  and  neck  shear. 

b.  The  crew  compartment (s)  can  be  instrumented  to  collect 
thermal,  toxic  fume,  and  blast  overpressure  data.  The  following 
crew  environment  data  have  been  collected  in  previous  LFTs: 

(1)  Temperature  and  heat  flux  levels  at  each  crew  member 
location.  These  data  allow  a  determination  of  the  level  of  burn 
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damage  and  the  effectiveness  of  the  crew  member's  protective 
uniform. 


(2)  Toxic  fume  levels  at  each  crew  member  location.  Data 
on  toxic  gases,  pyrolysis  products,  and  airborne  particulates  are 
collected. 

(3)  Blast  overpressure  levels  at  various  crew  locations. 
These  data  are  used  to  determine  the  level  of  crew  incapacitation 
due  to  injury  to  the  air  containing  structures  of  the  body 
(e.g.,  lungs  and  ears). 

c.  The  collected  simulant  and  environmental  data  are  analyzed 
and  compared  to  approved  crew  injury  criteria  to  determine  an 
expected  level  of  crew  incapacitation.  These  data  are  used  by 
SLAD/BVLD  in  the  overall  crew  survivability  assessment. 

5-13.  Vehicle  Vulnerability 

a.  After  each  individual  shot,  all  damage  is  recorded,  as 
well  as  obvious  vehicle  functional  degradation  (e.g.,  engine  will 
not  run) .  This  damage  assessment  is  then  used  to  determine  vehicle 
vulnerability  in  the  form  of  single-shot  kill  estimates.  These 
estimates  are  derived  from  the  damage  assessment  report  by  the  use 
of  fault-tree  or  deactivation  diagrams  and  an  assessment,  by  the 
user,  of  the  resulting  LOF.  Existing  kill  categories  for  armored 
vehicles  and  aircraft  systems  are  presented  below. 

(1)  Mobility  Kill  (M-kill) .  An  armored  vehicle  suffers  a 
M-kill  if  it  becomes  incapable  of  executing  controlled  movement  and 
cannot  be  repaired  by  the  crew  (within  approximately  ten  minutes) 
on  the  battlefield. 

(2)  Firepower  Kill  (F-kill) .  An  armored  vehicle  suffers  a 
F-kill  if  it  becomes  incapable  of  delivering  accurate,  controlled 
firepower  and  cannot  be  repaired  by  the  crew  (within  approximately 
ten  minutes)  on  the  battlefield. 

(3)  Catastrophic  Kill  (K-kill) .  An  armored  vehicle 
sustains  a  K-kill  when  both  a  M-kill  and  a  F-kill  occur  and  it  is 
not  economically  repairable. 

(4)  Attrition  Kill.  An  attrition  kill  is  obtained  when  an 
aircraft  sustains  combat  damage  so  extensive  that  it  is  neither 
reasonable  nor  economic  to  repair.  This  category  is  divided  into 
six  levels  of  kill  depending  on  the  time  after  impact  at  which 
manned  control  is  no  longer  achievable. 

(5)  Forced  Landing.  This  kill  is  obtained  when  an 
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aircraft  sustains  combat  damage  that  forces  the  crew  to  execute  a 
controlled  landing  (powered  or  unpowered) .  This  category  includes 
aircraft  which  will  require  repairs  for  flight  to  another  area  and 
aircraft  which  cannot  be  repaired  on-site  but  can  be  recovered  by  a 
special  team. 

(6)  Mission  Abort.  This  kill  is  obtained  when  an  aircraft 
sustains  combat  damage  that  prevents  completion  of  the  designated 
mission  but  permits  the  aircraft  to  return  to  base. 

(7)  Mission  Available.  This  kill  is  obtained  when  an 
aircraft  has  landed  but  will  require  repair  before  returning  to  a 
mission-ready  status. 

b.  In  addition  to  providing  insights  into  system 
vulnerability,  LFT&E  can  provide  the  user  "hands-on”  experience  in 
BDAR.  During  LFT&E,  BDAR  can  provide  the  user  insights  into  the 
time,  parts,  tools,  and  skills  required  to  repair  the  system  to  a 
combat— capable  condition.  Evaluation  of  a  system's  capabilities 
immediately  following  a  simulated  threat  attack  compared  to  the 
system's  capabilities  following  crew,  organizational,  and  direct 
support  repair  provides  insights  into  the  overall  fightability  of 
the  system.  Another  application  of  the  repair  process  is  to 
examine  the  spare  part  supply  line  to  ensure  parts  stocked  are  in 
fact  those  required  to  support  damage  sustained  from  a  battlefield 
encounter . 
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Chapter  6 
Lessons  Learned 


Section  I 
Introduction 


6-1.  General 

Live  Fire  Testing  is  one  of  the  most  visible  and  expensive  phases 
of  developmental  testing  and  requires  detailed  planning, 
documentation,  and  coordination  in  order  to  ensure  an  efficient 
and  effective  program.  To  make  it  affordable  and  efficient  and 
to  ensure  that  the  Army  is  provided  the  best  information  for  its 
investment,  future  LFT&E  efforts  must  take  advantage  of 
experience  gained  during  the  development  and  conduct  of  previous 
LFT&E  programs.  Incorporation  of  these  lessons  into  the  planning 
and  conduct  of  future  LFT&E  efforts  will  ensure  that  the  maximum 
return  is  achieved  on  the  Army's  investment  in  LFT&E. 

6-2.  Affordable 

To  keep  LFT&E  affordable,  the  number  of  full-up  shots  must  be 
kept  to  an  absolute  minimum.  To  minimize  the  number  of  full-up 
system  shots,  the  LFT&E  strategy  must  be  structured  with 
extensive  component— level  tests  which  collectively  support 
resolution  of  critical  system  survivability  and/or  lethality 
issues.  A  successful  component-level  test  program  can  minimize 
the  number  of  required  full-up  shots  and  thereby  reduce  LFT 
costs.  The  information  from  a  few,  well-designed  full-up  shots 
can  lead  to  system  vulnerability  reductions  or  lethality 
enhancements  which  can  make  a  difference  in  the  battlefield 
survivability  of  both  the  crew  and  the  system. 


Section  II 

Test  Planning  and  Execution 


Recently,  the  Army  conducted  extensive  LFTs  on  its  two  primary 
combat  vehicles  -  the  Abrams  Tank  and  the  Bradley  Fighting 
Vehicle.  This  paragraph  summarizes  key  lessons  learned  from  a 
test  planning  and  test  execution  standpoint  which  are  applicable 
to  future  vulnerability  and  lethality  LFTs. 

6-3 .  Management  .  .  ^ . 

Management  of  a  LFT  is  best  accomplished  by  a  matrix  organization 
responsible  for  test  planning,  test  execution,  evaluation  of  test 
results,  and  test  documentation.  That  organization  must  have 
access  to  professionals  with  expertise  in  ballistics, 
vulnerability/ lethality  modeling,  casualty  and  damage  assessment, 
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developmental  testing,  combat  vehicle  repair,  BDAR,  and  materiel 
systems  analysis  and  evaluation.  Figure  6-1  illustrates  the 
matrix  organization  created  by  the  Army  for  the  conduct  of  the 
Bradley  and  Abrams  LFTs. 

6-4 .  Documentation 

Prior  to  firing  a  round,  prepare  an  lEP/TDP,  a  DTP,  and  obtain 
approval /comments  from  both  the  Army  and  OSD  leadership.  Include 
in  the  lEP/TDP  and  the  DTP  all  information  required  by  the  OSD 
LFT&E  Guidelines  and  ensure  all  testing  and  subsequent 
evaluations  are  conducted  in  strict  compliance  with  these  plans. 
These  plans  must  be  of  sufficient  detail  to  preclude 
misunderstanding  by  the  Army  and  OSD  leadership. 

6-5.  Participants 

Identify  all  oversight  organizations  (e.g.,  OSD,  General 
Accounting  Office,  etc.)  and  involve  them  both  in  the  test  design 
and  test  execution  processes.  Concerns  by  these  organizations 
must  be  raised  and  addressed  prior  to  or  during  testing  -  NOT 
after  test  completion. 

6-6.  Data  Sharing 

Share  test  data  with  all  activities  as  soon  as  data  are  validated 
so  that  oversight  organizations,  the  independent  evaluator,  the 
PM,  and  the  user  representatives  have  the  same  information  at  any 
point  during  the  test. 

6-7.  Emerging  Results 

Establish  a  formalized  emerging  results  forum  where  the  matrix 
test  team  can  identify  and  document  potential  evaluation  issues. 
Follow-up  with  research  by  the  PM  and/or  off-line  tests  and 
investigations  by  the  system  contractor  to  shed  additional  light 
on  these  potential  issues  so  that  they  are  either  resolved  prior 
to  test  completion  or  identified  for  inclusion  in  the  final 
evaluation  report.  The  guiding  principle  must  be  to  address  all 
vulnerability/lethality  concerns  in  the  emerging  results  forum  as 
they  are  identified  to  preclude  ••surprises”  by  the  contents  of 
the  final  reports.  The  OSD  will  be  provided  access  to  the 
emerging  results.  However,  all  emerging  results  identified  by 
OSD  for  use  in  supporting  its  independent  assessment  will  be 
provided  the  Army  for  factual  review  prior  to  its  use. 

6-8.  Evaluation 

Prepare  a  balanced  evaluation  report  which  objectively  describes 
both  the  negative  and  positive  aspects  of  the  results.  For 
example,  not  all  vulnerabilities  identified  in  a  vulnerability 
LFT&E  can  be  fixed.  Constraints  on  system  funding,  system 
weight,  etc.,  necessitate  that  the  matrix  team  participate  in  the 
prioritization  of  the  identified  vulnerabilities  from  the 
perspectives  of  likelihood  of  occurrence  on  the  battlefield  and 
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the  degree  of  system  degradation  given  an  occurrence.  The  final 
evaluation  report  provides  that  information  to  the  user  and  to 
the  PM  for  resolution. 

6-8.  Evaluation 

Prepare  a  balanced  evaluation  report  which  objectively  describes 
both  the  negative  and  positive  aspects  of  the  results.  For 
example,  not  all  vulnerabilities  identified  in  a  vulnerability 
LFT&E  can  be  fixed.  Constraints  on  system  funding,  system 
weight,  etc.,  necessitate  that  the  matrix  team  participate  in  the 
prioritization  of  the  identified  vulnerabilities  from  the 
perspectives  of  likelihood  of  occurrence  on  the  battlefield  and 
the  degree  of  system  degradation  given  an  occurrence.  The  final 
evaluation  report  provides  that  information  to  the  user  and  to 
the  PM  for  resolution. 


Section  III 

Technical  Lessons  Learned 


6-9 .  Lessons  Learned 

During  the  conduct  of  previous  LFT&Es,  lessons  learned  from  a 
technical  aspect  were  identified  and  are  enumerated  below.  These 
technical  lessons  learned  not  only  address  LFT&E  but  also  weapon 
system  design  principles  which  may  prove  useful  in  future 
development  endeavors. 

a.  Live  Fire  Testing  is  beneficial.  Extensive  use  of  off¬ 
line  testing  to  address  such  issues  as  sympathetic  detonation, 
ammunition/propellant  compartmentalization,  and  collateral  damage 
will  keep  LFT&E  affordable  while  ensuring  that  critical  issues 
are  adequately  addressed.  Preserve  valuable  assets  to  address 
those  issues  which  require  a  full-up  system  to  answer. 

b.  Identify  vulnerability  reduction/ lethality  enhancement 
measures  early  in  the  development  process.  Take  advantage  of  the 
SLAD/BVLD  expertise  in  this  area.  Do  not  wait  until  LFT  to 
ensure  vulnerability  reduction/ lethality  enhancement  measures  are 
adequate.  Vulnerability  reduction/ lethality  enhancement  measures 
must  be  considered  during  the  system  design  process. 

c.  Design  the  system  for  both  peacetime  and  wartime  use.  For 
example,  the  Abrams  Tank  was  developed  with  an  inhibitor  which 
would  not  allow  the  driver  to  exceed  two  miles  per  hour  if  the 
system  indicated  the  engine  had  been  damaged.  Obviously,  this 
was  to  preserve  valuable  engines  during  peacetime 
training/testing  operation  but  can  significantly  limit  vehicle 
performance  during  combat  situations.  Thus,  "combat  overrides" 
to  "peacetime  inhibitors"  should  be  a  principal  design 
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consideration.  (NOTE:  The  problem  cited  in  this  example  was 
identified  during  Abrams  Vulnerability  LFT;  the  PM  has  developed 
a  fix  which  corrects  this  problem.) 

d.  Take  a  total  systems  look  at  crew  and  system 
vulnerability.  This  means  one  must  consider  the  contribution  of 
all  items  (crew  clothing,  component  hardware,  ammunition,  fuel, 
and  stowage  items)  to  crew  and  system  vulnerability.  For 
example,  design  or  store  stowage  items  so  that  they  do  not  pose  a 
fire  hazard  to  the  crew. 
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Figure  6-1.  Bradley  and  Abrams  LFT  Matrix  Organisation 
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Chapter  7 

Alternatives  for  Threat  Tank  and  Helicopter  Targets 


Section  I 
Introduction 

7-1.  Source  of  Information 

This  information  in  this  chapter  is  taken  (almost  verbatim)  from 
two  AMSAA  memoranda  (14  March  1989  and  19  May  1989)  on  the 
subject  of  Live  Fire  Lethality  Test  Target  Surrogates. 


Section  II 

Tank  Targets  for  Live  Fire  Lethality  Testing 


7-2.  General 

Live  Fire  Lethality  testing  of  U.S.  antitank  munitions  shall 
include,  when  feasible,  the  firing  of  that  munition  against 
threat  targets.  Since  it  is  very  unlikely  that  future  threat 
tank  targets  (requirements  for  antitank  munitions)  will  be^ 
available  at  the  time  they  are  needed  for  Live  Fire  Lethality 
testing,  it  is  necessary  to  identify  and  evaluate  alternatives 
for  threat  tank  targets  and  recommend  the  preferred 
alternative (s) . 

7-3.  Classification  Scheme 

The  Army  DCSINT  has  developed  a  classification  scheme  for  threat 
tank  range  target  (armor  arrays)  data.  It  is  also  applicable  to 
tank  target  (reference  target)  data.  The  categories  are: 

a.  An  original  or  actual  specimen. 

b.  A  duplicate  or  replica  created  from  original 
specification . 

c.  A  surrogate  or  reasonable  facsimile  which  is  created  from 
specific  knowledge  about  original  specifications. 

d.  A  substitute  which  represents  some  general  knowledge  or 
performance  characteristics  of  the  original. 

e.  A  postulated  technology  option  derived  from  an 
intelligence  assessment. 

7-4.  Threat  Tank  Target  Data 

For  Live  Fire  Lethality  testing  of  antitank  munitions  the  threat 
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tank  target  data  are  and  will  probably  always  be  a  postulated 
technology  option  derived  from  an  intelligence  assessment. 

7-5.  LFTE  Alternatives 

a.  The  testing  and  evaluation  alternatives  are  defined 
(Table  7-1)  in  terms  of  the  type  of  target  fired  at  in  Live  Fire 
Lethality  tests,  whether  the  target  functions  (i.e.,  mobility, 
firepower,  etc.),  what  the  test  addresses  (armor  perforation, 
damage  mechanisms,  components) ,  and  the  basis  for  the  overall 
lethality  assessment  (test,  model) . 

(INSERT  TABLE  7-1) 

b.  The  eight  lethality  test  and  evaluation  alternatives  break 
logically  into  three  groups: 

(1)  Functioning  tanks  with  an  overall  lethality  assessment 
based  upon  test  results  (alternatives  1-4) . 

(2)  Ballistic  hull  and  turret  with  the  crew  (alternative 
6)  or  crew  and  components  (alternative  5)  represented  by  boxes 
with  a  limited  overall  lethality  assessment  based  upon  test 
results. 

(3)  Ballistic  hull  and  turret  only  (alternative  7)  or 
range  targets  (alternative  8)  with  no  overall  lethality 
assessment  based  upon  test  results. 

7-6.  Modeling 

Regardless  of  the  scope  of  lethality  testing,  the  overall 
lethality  assessment  will  be  supplemented  with  model  predictions. 

7-7.  Antitank  Damage  Mechanisms 

a.  Live  fire  and  joint  live  fire  lethality  and  vulnerability 
testing  indicates  that  the  dominant  antitank  damage  mechanisms 
are  the  primary  penetrator  and  spall.  Lethality  of  antitank 
projectiles  depends  on  whether  the  armor  is  perforated,  the 
extent  of  spall  damage  (cone  angle  and  amount  of  spall) ,  the 
location  of  major  vulnerable  components  (crew,  ammo,  fuel,  etc.), 
and  the  specific  way  the  tank  design  is  implemented  (e.g.,  do 
subsystems  fail  gracefully,  location  and  distribution  of 
hydraulic  and  electrical  power  lines,  are  there  redundant 
systems) . 

b.  Spall  damage  varies  as  a  function  of  the  type  of  armor 
perforated  (e.g.,  laminate  ceramic  versus  reactive)  because  the 
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penetrator  is  attacked  differently  by  the  different  armor  defeat 
mechanisms.  A  penetrator  could  have  the  same  degree  of  overmatch 
against  two  different  armor  technologies  and  have  different  spall 
characteristics . 


c.  In  addition,  the  target  structure  and  way  the  armor  is 
integrated  into  the  vehicle  structure  will  affect  collateral 
damage  to  components  (e.g.,  damage  due  to  ballistic  shock)  and 
affect  multi  hit  performance  of  the  vehicle.  Technical 
projections  of,  future  tanks  address  armor  and  gross 
characteristics.  However,  the  details  on  the  components  and 
vehicle  design  cannot  be  adequately  projected. 

7-8 .  Target  Alternatives 

a.  Functioning  tank  alternatives  are  appealing  because 
lethality  can  be  assessed  directly  for  each  shot  fired.  The 
practical  limits  on  numbers  of  shots  fired  dictates  that  an 
overall  lethality  assessment  be  based  upon  both  test  results  and 
calculations  for  a  broader  spectrum  of  conditions.  Ideally,  the 
testing  should  confirm  that  the  model  predictions  are  accurate  or 
provide  the  basis  for  modifying  the  model  to  permit  an  accurate 
set  of  predictions. 

b.  Alternatives  1  and  2  have  the  highest  level  of  perceived 
fidelity  because  the  armor  and  the  configuration  are  matching  or 
trying  to  match  the  threat  projections.  Actual  fidelity  will 
probably  be  considerably  less  than  perceived  fidelity  because 
there  is  little  or  no  information  on  which  to  project  component 
characteristics  and  vehicle  design.  It  is  very  unlikely  that  the 
component  vulnerability  or  the  system  loss  of  function  for  these 
targets  will  match  the  future  threat. 

c.  In  addition,  the  configuration  projections  for  the  future 
threat  tanks  are  radically  different  than  any  available  U.S.  or 
older  threat  systems,  and  it  is  unlikely  that  these  kinds  of 
modifications  could  be  made  in  a  configuration  with  acceptable 
functioning  or  vulnerability  characteristics.  Therefore,  neither 
alternatives  1  or  2  are  recommended. 

d.  Armor  on  U.S.  or  available  older  threat  tanks  can  be 
modified  to  represent  a  level  of  overmatch  (residual  penetrating 
capability)  or  can  be  replaced  with  a  range  target  that 
represents  the  threat  armor  projections.  Both  options  have 
potentially  significant  limitations.  Modifying  the  armor  to 
represent  a  level  of  overmatch  probably  would  not  represent  spall 
accurately.  Replacing  the  armor  with  a  range  target  may  alter 
significantly  the  collateral  damage  to  components  and,  therefore. 
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may  give  a  misleading  impression  of  system  loss  of  function. 

Given  these  limitations  and  since  the  configurations  of  future 
threat  tanks  are  significantly  different  than  the  U.S.  or  older 
available  threat  tanks,  alternative  3  is  not  recommended. 

e.  Alternative  4  (U.S.  or  older  threat  tank  without 
modifications)  is  not  recommended  because  it  does  not  match 
threat  projections  for  armor,  components,  or  configuration. 
However,  firing  against  a  U.S.  or  older  available  threat  tank 
without  modification  can  provide  some  useful  supplemental  insight 
into  the  overall  lethality  assessment  against  tank  targets.  In 
addition,  it  adds  to  the  vulnerability  data  base  on  these  tanks. 

f.  Ballistic  hull  and  turret  (BHT)  alternatives  5-7  built  to 
threat  armor  projections  can  provide  an  accurate  representation 
of  the  areas  that  can  be  perforated  and  the  degree  of  behind- 
armor  effects  for  areas  perforated.  If  components  (mannequins 
and  boxes  that  represent  components  occupy  the  projected  areas) 
are  included  then  a  limited  lethality  assessment  for  shots  fired 
based  directly  upon  tests  can  be  made.  Alternative  5  is  the  most 
comprehensive  of  these  alternatives,  but  it  is  not  sufficient 
because  the  target  does  not  function. 

g.  Alternative  8  is  not  recommended  because  it  does  not 
provide  even  a  limited  basis  for  assessing  lethality  against  the 
threat  tank  for  the  shots  fired  directly  from  the  test.  In 
addition,  firings  against  range  targets  have  always  been  part  of 
the  standard  development  tests  that  contribute  to  the  overall 
lethality  assessment. 

7-9.  Alternative  Selection 

None  of  the  alternatives  by  themselves  is  adequate  for  live  fire 
lethality  testing;  however,  it  is  possible  to  fire  against  three 
different  targets  to  adequately  demonstrate  lethality  in  live 
fire  tests.  The  three  different  targets  and  the  type  of  tests 
recommended  are  as  follows: 

a.  Threat  tank  range  target  tests  with  sufficient  sample 
sizes  to  establish  with  high  statistical  confidence  that  the 
ability  of  the  baseline  and  developmental  anti-armor  munition  to 
perforate  the  range  targets  of  interest  and  to  characterize  the 
behind-armor  spall  characteristics  of  both  munitions. 

b.  Ballistic  hull  and  turret  targets  constructed  to  threat 
armor  projections  and  configured  with  crew  and  major  component 
box  representations  to  demonstrate  major  lethality  differences 
between  baseline  and  developmental  anti-armor  munitions. 
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c.  U.S.  or  older  threat  tank  (without  modifications)  to 
provide  a  limited  demonstration  of  lethality  of  the  baseline  and 
developmental  munitions  against  a  functioning  vehicle.  (Note 
these  tests  may  not  demonstrate  significant  differences  because 
both  munitions  may  significantly  overmatch  these  targets.) 


Section  III 

Helicopter  Targets  for  Live  Fire  Lethality  Testing 


7-10.  General 

Live  fire  testing  of  U.S.  anti-helicopter  munitions  shall 
include,  when  feasible,  the  firing  of  that  munition  against 
threat  targets.  Since  it  is  very  unlikely  that  future  threat 
helicopter  targets  (requirements  for  antihelicopter  munitions) 
will  be  available  at  the  time  they  are  needed  for  Live  Fire 
Lethality  testing,  it  is  necessary  to  identify  and  evaluate 
alternatives  for  threat  helicopter  targets  and  recommend  the 
preferred  alternatives.  Thus,  for  Live  Fire  Lethality  testing  of 
antihelicopter  munitions  the  threat  helicopter  target  data  are 
and  will  probably  largely  be  postulated  technologies  and 
configurations  derived  from  intelligence  assessments. 

7-11.  Munitions-Target  Interactions 

Our  antihelicopter  munitions  will  be  largely  smart  munitions  in 
that  they  will  respond  to  target  signatures  to  execute  terminal 
maneuvers  (optimizing  accuracy  and  approach  angles)  or  optimize 
fuzing/ detonation  points  relative  to  the  target  by  sensing  target 
proximity.  These  complex  interactions  of  our  antihelicopter 
munitions  with  a  variety  of  signatures,  countermeasures  (false 
signatures  or  obscuration/suppression  of  signatures) ,  and 
environmental  conditions  will  require  substantive  laboratory, 
technical,  and  operational  testing  preceding  any  Live  Fire 
testing  so  that  Live  Fire  testers  have  sufficient  basis  to  define 
shots  about  meaningful  intercept  conditions. 

7-12.  Aircraft  Damage  Mechanisms 

a.  Lethality  and  vulnerability  testing  indicates  that  the 
dominant  damage  mechanisms  against  aircraft  (including 
helicopters)  are  the  warhead  fragments  and  blast.  The  blast 
effect  are  significantly  enhanced  by  warhead  penetration  of  the 
aircraft  skin.  Munitions  optimized  for  fragment  kills  use 
various  forms  or  fuzing  to  detonate  the  warhead  when  in  proximity 
of  the  threat  aircraft.  Munitions  which  depend  on  the  blast  kill 
mechanism  may  optimize  their  point  of  impact  based  on  target 
signature  to  take  advantage  of  vulnerability  features  of  the 
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aircraft.  Hit-to-kill  missiles  can  achieve  bonus  effects  by  the 
missile  body  continuing  to  fly  into  the  aircraft  target. 

b.  There  are  also  dual  purpose  munitions  which  are  designed 
for  both  anti-armor  and  anti-aircraft  roles.  Typically  these 
munitions  have  a  point  detonating  shaped  charge  warhead  matched 
to  the  armor  threat  of  interest  and  for  the  air  defense  role  a 
proximity  fuze  which  senses  if  the  munition  is  about  to  miss  the 
target  and  detonates  the  warhead  to  generate  lethal 
fragmentation.  The  shaped  charge  warhead  effect  is  often  so 
overmatched  to  the  aircraft  target  that  a  direct  hit  is  a  virtual 
kill. 

7-13.  Target  Alternatives 

Live  fire  testing  and  evaluation  alternatives  are  defined 
(Table  7-2)  in  terms  of  the  type  of  target  fired  at  in  Live  Fire 
Lethality  tests,  whether  the  target  is  functional  or  flyable, 
what  the  test  addresses,  and  the  basis  for  the  overall  lethality 
assessment  (test,  model,  or  a  combination  of  both) . 

(INSERT  TABLE  7-2) 

a.  Alternatives  1  and  2  are  certainly  the  most  credible 
alternatives  for  addressing  live  fire  objectives.  The 
acquisition  of  actual  threat  targets  in  quantity  for  exploitation 
(Alternative  1)  in  live  fire  will  not  likely  include  the  latest 
threat  systems.  It  may  be  possible  to  acquire  limited  numbers  of 
threat  helicopters  which  have  been  fielded  in  quantity  and 
exported  widely.  Also,  commercial  versions  of  some  helicopter 
types  may  be  available  on  the  world  market  as  the  Soviets  or 
others  seek  opportunities  to  gain  hard  currency. 

b.  The  construction  of  flyable  surrogates  based  bn  technical 
threat  projections  (Alternative  2)  is  a  very  costly  and  risky 
approach  to  satisfying  live  fire  target  requirements. 

Development  costs  would  be  similar  to  the  development  of  any  new 
helicopter  and  the  uncertainty  of  projections  carries  with  it  the 
risk  of  developing  exactly  the  wrong  surrogate.  However,  the 
development  of  non-flying/non-functioning  surrogate  targets  based 
on  threat  projections  is  probably  very  reasonable.  This  approach 
might  also  allow  the  exploration  of  projected  design  alternatives 
at  reasonable  costs. 

c.  The  use  of  modified  US  or  older  threat  aircraft 
(Alternative  3)  provides  the  most  reasonable  alternative  for 
conducting  Live  Fire  Tests  of  fully  functioning  helicopters.  Use 
of  helicopters  which  most  closely  conform  to  the  projected  threat 
design  can  assist  in  understanding  the  influence  of  dynamic 
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effects  on  munitions  lethality. 


d.  Finally,  the  construction  of  a  fuselage  shell,  major 
component  boxes  and  mannequins  (Alternative  4)  based  on  technical 
threat  projections,  although  significantly  lacking  in  resolution, 
may  be  the  solution  to  maximizing  the  utility  of  a  limited  number 
of  more  faithful  but  expensive  target  alternatives.  This 
alternative  should  allow  the  exploration  of  expected  effects 
through  numerous  live  fire  shots  and  permit  "tuning  of  our 
models  to  minimize  the  number  of  shots  required  against  the  more 
expensive  and  scarce  alternatives. 


7-14 .  Alternative  Categories  . 

The  lethality  test  and  evaluation  target  alternatives  break 
logically  into  four  categories.  None  of  these  categories  offer 
an  entirely  satisfactory  technical  solution  and  the  costs 
associated  with  flying  and  functioning  targets  are  very  high. 

The  categories  for  each  alternative  are: 

a.  Flyable-functioning  helicopters  with  an  overall  lethality 
assessment  primarily  based  upon  observation  of  test  results 
(collection  of  damage  effects  data  frequently  not  possible) . 


b.  Non-flyable/ functioning  helicopter  targets  (typically 
tower  mounted)  with  an  overall  lethality  assessment  based  upon  a 
combination  of  modelling  (principally  to  define  intercept  and 
fuzing/detonation  points)  and  test  results  (collection  of  damage 

effects  data) . 


c.  Non-f lyable/nonfunctioning  helicopter  targets  (typically 
tower  mounted)  with  an  overall  lethality  assessment  based  upon  a 
combination  of  modelling  (defining  intercept/ fuzing  and  damage 
effects  on  nonfunctioning  components)  and  test  results 
(collection  of  damage  effects  data) . 


d.  Fuselage  or  major  subsystems  representative  of  comparable 
threat  helicopter  components  (i.e.,  engine,  rotor  system,  etc.), 
with  an  overall  lethality  assessment  based  principally  upon 
modelling. 


7-15.  Category  Selection 

a.  The  most  realistic  category  for  live  fire  lethality 
against  threat  helicopters  would  appear  to  be  against  the  flying 
helicopter  targets  (drone  kits  installed  and  flown  by  a  remote 
flight  qualified  operator) .  However,  using  flying  tarpts  alone 
is  not  a  totally  acceptable  solution  with  regard  to  data 
collection. 
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b.  A  lethal  engagement  of  a  flying  drone  threat  helicopter  or 
surrogate  will  most  often  result  in  a  crash  which  causes  other 
damage  tending  to  mask  damage  effects  created  by  the  weapon  fired 
at  the  aircraft.  True  lethality  might  also  be  masked  by  "lethal” 
damage  to  the  drone  control  system  resulting  in  a  helicopter 
target  crash. 

c.  Diagnosis  of  which  effects  are  munition  related  or  crash 
related  in  determining  lethality  for  live  fire  is  costly  and  time 
consuming,  and  often  not  possible.  Under  these  conditions, 
refurbishment  of  targets  may  frequently  be  out  of  the  question. 

d.  This  target  category  may  require  nearly  one  target  per 
engagement  fired  for  live  fire  lethality  (at  possible  millions  of 
dollars  per  copy  depending  upon  the  fidelity  of  representing  the 
threat) . 

e.  The  next  most  realistic  category  for  live  fire  lethality 
testing  would  be  against  non-flying/functioning  alternatives. 
Helicopters  can  be  tied  down  on  a  tower  and  simulate  a  hovering 
or  slow  flying  helicopter  by  running  engines,  rotor  blades,  and 
other  subsystems  to  create  conditions  for  obtaining  fairly 
realistic  lethality  effects  given  that  intercept  and  fuzing 
geometries  are  well  understood. 

f.  Although  damage  effects  can  be  acquired  from  functioning 
tower  mounted  targets,  the  true  test  of  whether  or  not  the 
helicopter  would  have  crashed  from  some  damage  effects  in  dynamic 
flight  may  only  be  derived  with  absolute  certainty  by  using  a 
flying  helicopter.  For  example,  rotor  blades  hit  while  rotating 
on  a  tethered  machine  are  subject  to  different  dynamic  loads  than 
rotor  blades  on  a  real  hovering  helicopter  and  the  effects  on 
controlled  flight  may  not  be  readily  apparent  in  the  tethered 
case. 

g.  Functioning  (flying  or  non-flying)  helicopter  alternatives 
are  most  appealing  because  nearly  the  entire  range  of  damage 
effects  may  be  assessed  for  each  shot. 

h.  The  non-flying/non-functioning  category  of  threat  target 
is  a  less  realistic  category  for  each  alternative,  but  is 
probably  one  of  the  more  practical  and  among  the  most  affordable 
of  the  categories  for  each  alternative.  However,  without  engines 
running  and  rotor  blades  rotating,  the  realism  of  live  fire 
lethality  will  be  limited  for  some  munition  effects. 

i.  The  least  realistic  category  for  each  alternative  is  the 
use  of  the  fuselage  and/or  major  components  for  live  fire 
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testing.  However,  from  a  practical  point  of  view  the  fuselage 
and  component  shots  are  an  essential  building  block  to  gaining 
enough  information  to  tune  lethality  modelling  and  decide  on  the 
nature  of  "full  up"  shots  against  more  realistic  target 
alternatives . 

7-16 .  LFTE  Approach 

a.  The  major  thesis  to  be  derived  from  the  preceding 
discussion  is  to  proceed  on  a  building  block  approach  from  the 
crudest  representation  of  threat  targets  to  build  a 
vulnerability /lethality  data  base  and  to  minimize  the  use  of 
scarce  and  costly  threat  target  resources  in  the  live  fire 
program.  It  is  probably  not  reasonable  to  expect  a  cookbook 
solution  for  all  types  of  munitions. 

b.  The  preceding  discussion  may  offer  a  hierarchy  of 
solutions  where  we  might  enter  the  Table  1  matrix  at  the 
component  level  in  developing  basic  vulnerability  assessments  and 
then  developing  our  live  fire  lethality  data  using  helicopter 
targets  which  provide  acceptable  levels  of  realism  and 
affordability. 

c.  A  combination  of  Alternatives  2c/d  (non-f lyable/non- 
functioning  and  fuselage-major  subsystem  surrogates)  and  3a/b 
(flyable-functioning  and  non-f lyable/functioning  older  US  or 
threat  surrogates)  is  probably  the  most  practical  and  affordable 
approach  for  building  a  realistic  matrix  of  live  fire  lethality 
tests  against  threat  helicopters. 
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Chaptar  i  Army  Softwara  Tast  and  Evaluation 

Saction  I 
Introduction 

1-1.  Scopa 

This  material  contained  herein  applies  to  all  Army 
software  TiE  performed  for  systems  developed  and/or 
maintained  in  accordance  with  (lAW)  AR  70,  Materiel 
System  Computer  Resources  (MSCR) ,  and  AR  25,  Automated 
Information  Systems  (AIS) ,  series  of  regulations.  This 
includes  TiE  in  relation  to  development  and  support  of 
the  software  element  of  firmware.  It  provides  both 
DOD-STD-2167A  and  DOD-STD-7935A  documentation  profiles. 

1-2.  Objectives 

This  part  of  DA  Pamphlet  73-1: 

a.  Describes  a  unified  software  test  and  evaluation 
process  for  MSCR  and  AIS. 

b.  Describes  the  implementation  procedures  for  AR 
73-1  policies  related  to  software  T&E. 

c.  Describes  a  disciplined  approach  to  life  cycle 
software  T&E. 

d.  Describes  the  Army  standard  for  planning  and 
implementing  software  test  and  evaluation.  This  will 
promote : 

(1)  Consistency  and  ease  of  application. 

(2)  Early  involvement  of  the  T&E  community  in 
the  acquisition  process. 

(3)  Demonstration  of  software  capabilities. 

(4)  Acquisition  process  improvements. 

Section  II 

Policy  Basis  of  Software  Test  and  Evaluation 
1-3.  General 

Army  software  test  and  evaluation  processes  and 
practices  have  evolved  over  the  years,  responding  to 
new  technologies,  resour  constraints,  organization 
changes,  and  lessons  les  -sd.  These  processes  and 
practices  require  all  ArEy  systems  involving  software, 
whether  MSCR  or  AIS,  to  undergo  continuous  product 
evaluation  throughout  the  life  cycle.  DOD  policies  for 
AIS  are  described  in  DOD  series  5000  and  7920,  and  DOD 
policies  for  MSCR  are  described  in  DOD  series  5000. 

DOD  policy  provides  the  basis  for  Army  T&E  policy  and 
procedures.  The  procurement  cycle  milestones  (MS)  and 
life  cycle  phases  described  in  this  part  of  DA  Pamphlet 
73-1  are  summarized  in  Figure  1-1. 

1-4.  Department  of  Defense  Directive  (DODD)  5000.1 
DODD  5000.1,  “Defense  Acquisition",  mandates  an 
integrated  framework  for  translating  broadly-stated 
mission  needs  into  stable,  affordable  acquisition 
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programs  that  meet  the .user’s  needs  and  can  be 
sustained  given  projected  resources,  it  also  calls  for 
a  rigorous,  event“oriented  management  process  for 
acguiring  guality  products  that  emphasize  effective 
acquisition  planning  and  improved  communications  with 
the  users.  Milestones  and  decision  events  are 
identified.  Test  and  evaluation  shall  be  used  to 
determine  system  maturity  and  identify  areas  of 
technical  risk.  Further,  the  directive  indicates  that 
performance  objectives  must  satisfy  identified 
operational  needs  and  be  verifiable  by  testing. 
Performance  objectives  .include  critical  supportabilitv 
factors  such  as  reliability,  availability,  and 
maintainability.  DODD  5000.1  also  mandates  independent 
operational  test  activities.  ^ 


1-5.  Department  of  Defense  Instruction  (DODZ)  5000.2 

DODI  5000.2,  "Defense  Acquisition  Management  Policies 
and  Procedures"  and  its  accompanying  manuals,  provide 
the  implementation  for  DODD  5000.1.  Software  and 
system  T&E  must  address: 

achievement  of  technical  performance 
specifications  and  objectives.  ^ 

.  Establishing  exit  criteria  for  results  that  must 

be  attained.  These  can  be  viewed  as  gates  through 
which  programs  must  pass  during  each  phase. 

c.  Ensuring  systems  are  operationally  effective  and 
suitable. 

d.  Providing  essential  information  to  support 
decision-making. 
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e.  Designing  test  objectives  to  demonstrate 
capabilities  appropriate  to  phase  and  milestone. 

f.  Utilizing  early  operational  assessments  and 
early  prototype  testing. 

g.  Utilizing  early  test  planning. 

h.  Preparing  test  and  evaluation  master  plans 
(TEMPS.) 

i.  Obtaining  formal  developer  certification  of 
readiness  for  operational  testing. 

j.  Restricting  system  contractor  participation  in 
operational  T&E  supporting  a  Milestone  III  production 
decision. 

)c.  Managing  computer  resources  development  as  an 
integral  part  of  system  development. 

l.  Ensuring  software  maintainability  is  an  issue 
during  development. 

m.  Utilizing  a  disciplined  software  development 
process  based  on  effective  engineering  approaches. 

n.  Utilizing  Ada  as  the  only  programming  language 
for  new  defense  systems  and  major  upgrades  of  existing 
systems . 

o.  Ensuring  sc ftware  management  indicators  and 
metrics  will  be  u-  sd  in  management  of  software  effort. 

p.  Ensuring  software  engineering  practices  include: 

(1)  Rigorous  configuration  control  and  quality 
assurance  as  required  by  DOD-STD-2168. 

(2)  Walk-throughs,  inspections,  or  reviews  of 
requirements  documents,  design,  and  code. 

(3)  Formal  definition  and  deployment  of  quality 
control  procedures  and  milestone  quality  criteria. 

(4)  Software  security  and  virus  protection. 

(5)  Independent  verification  and  validation. 

(6)  Rigorous  testing  of  modules  and  interfaces 
at  all  levels  of  aggregation. 

q.  Emphasizing  analysis  of  software  errors  by 
contractor/developer  to  give  insight  into  reliability, 
quality,  safety,  cost  and  schedule  problems. 

r.  Utilizing  prototyping  and  models  from  H  astone 
0  to  Milestone  II  to  establish  and  refine  requ  ments. 
Use  of  prototyping  between  Milestone  II  and  Hi  .s 
permitted  if  areas  are  particularly  high  risk. 

1-6.  Department  of  Defense  Directive  (DODD)  7920.1 
DODD  7920.1,  "Life  Cycle  Management  of  Automated 
Information  Systems  (AISs),"  mandates  the  life  cycle 
phases  and  milestones  for  AIS  systems. 

1-7.  Department  Of  Defense  Instruction  (DODI)  7920.2 

DODI  7920.2  and  its  associated  manuals  provide 
procedures  for  implementing  DODD  7920.1.  It  mandates  a 
process  of  AIS  life  cycle  phases  and  associated 
products  which  encompass: 

(1)  Identification  of  needs 

(2)  Concept  development 
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(3)  Test  and  evaluation  planning 

(4)  Preparation  of  TEMPs 

(5)  Auditing  and  design  reviews 

(6)  Configuration  control 

(7)  Tracing  technical  specifications  to  mission 
need  and  prioritized  functional  requirements 

(8)  Quality  assurance  plans 

(9)  System  integration  plans 

(10)  Software  and  database  testing 

(11)  Malfunction  correction 

(12)  Operational  testing 

(13)  Effectiveness  and  suitability  of  training 

(14)  Logistics  support 

(15)  Continuity  of  operations 

(16)  Post-deployment  activities  and  assessments. 

1-8.  AR  70-1  "Systems  Acquisition  Policy  and 
Procedures" 

AR  70-1  covers  Army  policies  and  procedures  for 
materiel  system  acquisition  programs  and  implements 
DODD  5000.1  and  DODI  5000.2.  It  describes  policies  for 
MSCR  development,  documentation,  and  support.  It  also 
establishes  life  cycle  software  engineering  centers  for 
software  engineering  support  through  development  and 
during  post-deployment  software  support  (PDSS) .  The 
regulation  requires  new  mission-critical  software  to  be 

acquired  and  supported  lAW  D0D-STD-2167A,  DOD-STD- - 

and  DOD-STD-1467 .  It  establishes  the  compute'' 
resources  working  group  (CRWG)  and  the  comput-. 
resources  management  plan  (CRMP)  for  all  AR  70  systems. 

1-9.  AR  73-1  "Test  and  Evaluation  Policy" 

AR  73-1  covers  policies  for  the  Army's  test  and 
evaluation  program.  It  applies  to  all  systems  under 
the  auspices  of  70-series  and  25-series  regulations. 

It  defines  the  role  of  the  Army  Test  and  Evaluation 
Committee,  implements  the  Army's  continuous  evaluation 
program,  defines  the  role  of  the  independent 
evaluators,  and  includes  policies  for  the  TEMP. 

1-10.  AR  25-3  "Army  Life  Cycle  Management  of 
Information  Systems" 

AR  25—3  prescribes  policies  and  responsibilities  for 
the  life  cycle  management  of  information  systems  (ISs) 
as  described  in  AR  25-1.  It  implements  Office  of 
Management  and  Budget  (0MB)  Circular  A-109,  DODD 
7920.1,  DODI  7920.2  and  DODI  7920.4.  It  covers  need 
justification,  concept  development,  design, 
development,  acquisition,  test,  evaluation,  deployment, 
operation,  maintenance,  and  termination  of  life  cycle 
management  activities. 
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Saction  III 
Continuous  Evaluation 

1-11*  Background 

Continuous  evaluation  (CE)  is  fundamental  to  the  proper 
management  of  a  system.  Decision-making  must  be  based 
upon  substantive  evaluations  of  software 
characteristics,  and  maturity  and  reliability 
indicators  throughout  the  life  cycle.  These 
evaluations  are  paramount  to  producing  quality  software 
which  meets  the  user's  needs.  Failure  to  perform 
objective  evaluations  throughout  the  life  cycle  has 
resulted  in  significant  software  deficiencies,  system 
delays,  and  cost  overruns. 

1-12.  Definition 

Per  AR  73-1,  "Continuous  evaluation  is  a  process  which 
provides  a  continuous  flow  of  T&E  information  on  system 
status  and  will’ be  employed  on  all  acquisition 
programs."  Through  analysis  of  available  data,  CE 
assesses  technical  and  operational  performance, 
functionality,  effectiveness,  suitabilitv^ 
maintainability,  supportability,  and  id  tifies  risks. 
Software  CE  is  performed  by  software  er  neering 
personnel,  software  quality  assurance  (SOA)  personnel, 
independent  verification  and  validation  (IV&V) 
personnel,  and  developers,  as  well  as  Government 
developmental  and  operational  evaluation  personnel. 

1-13.  Objective 

The  objective  of  software  CE  is  to  provide  assessments 
of  software  and  system  progress.  It  is  a  form  of  risk 
analysis. 

1-14.  Scope 

CE  begins  during  the  mission  need  determination  and 
concept  exploration/definition  phases  and  continues 
through  post-deployment  support  to  system  retirement. 

CE  activities  are  tailored  to  the  scope  and  cost  of 
^®velopment  or  post  deployment  support  activities. 

1-15.  CB  activities  and  levels  of  evaluation 

a.  CE  activities  are  conducted  at  three  levels: 

(1)  CE  activities  performed  by  individuals  in 
the  development  organization.  The  goal  is  to  directly 
assess  and  impact  quality  through  development 
activities.  The  results  of  these  evaluations  are 
reported  within  these  organizations  to  document, 
correct,  and  re-examine  deficiencies.  Development 
organizations  are  those  contractors  or  Government 
agencies  tasked  to  perform  software  development  or 
maintenance  for  a  system. 

(2)  CE  activities  performed  by  individuals 
associated  with  the  project /program/product  manager 
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(PM)  such  as  SQA  and  other  matrix  support  personnel. 

The  goal  is  to  assess  software  and  system  quality, 
confopance  to  specifications,  maturity,  reliability, 
and  stability.  Eyaluations  performed  within  the  matrix 
support  organizations  are  reported  to  the  PM,  the 
developer,  and  independent  testers  and  evaluators. 

(3)  CE  activities  performed  by  Government  ■ 
developmental  and  operational  evaluators/assessors  lAW 
^  ^3-1.  The  goal  is  to  formally  assess  software  and 
system  acceptability  with  respect  to  operational 
effectiveness  and  suitability  for  deployment  and  use. 
Independent  ey^uators  report  results  to  the 
acquisition  conBhunity  (e.g.,  PM,  program  executive 
officers  (PEGS))  and  decision-makers  (e.g.,  Major 
Automated  Information  Systems  Review  Council  (MAISRC) ) . 

b.  CE  uses  all  available  data  sources  and  relies  on 
the  sharing  of  that  data.  It  is  imperative  that  the 
resups  of  all  evaluations  be  shared  among  the  various 
participating  communities  to  prevent  duplication, 
ensure  deficiencies  are  identified,  and  corrective 
actions  are  effective.  CE  consists  of  activities  such 
as: 


(1)  Collecting  and  analyzing  data  on  the 
software  engineering  processes  and  procedures  (e.g., 
requirements  analysis,  requirements  walk-through  and 
reviews,  design  procedures,  quality  control  procedures, 
prmal  reviews  and  audits,  and  testing) .  This  analysis 
identifies  processes  and  procedures  which  permit  poor 
quality  products  to  be  developed.  These  processes  and 
procedures  can  then  be  modified  or  replaced. 

(2)  Collecting  and  analyzing  data  on  the  ' 
software  products  and  the  results  of  events  (e.g., 
®y®^®®/®'J^system  specifications,  system  design  reviews, 
code  inspections) .  This  analysis  provides  indicators 
of  the  software/system  progress  and  health  (i.e., 
quality,  reliability,  stability,  and  maintainability) . 

(2)  Collecting  and  analyzing  data  on  corrective 
actions  to  ensure  proper  resolution. 

..w  evaluation  required  for  each  system 

throughout  its  life  cycle  is  determined  by  the 
acquisition  category,  the  cost  and  type  of  dollars 
emended  (i.e.,  operations  and  maintenance.  Army  (OMA) , 
other  procurement.  Army  (OPA) ,  research,  development, 
test  and  evaluation  (RDTE) ) ,  and  oversight  interest. 

d.  All  systems  must  be  evaluated/assessed  and  the 
results  reported  lAW  specified  criteria.  System 
acquisition  categories  and  classes  are  described  in 
Table  1-1.  Systems  requiring  independent  evaluation 

Table  1-2.  Acronyms  used  in  these  tables 
that  have  not  been  defined  as  yet  are  identified  below. 
They  are  addressed  in  detail  later  in  this  part  of  DA 
Pamphlet  73-1. 
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(1)  DAB  -  Defense  Acquisition  Board 

(2)  DISC4  -  Director  of  Information  Systems 

Command,  Control,  Communications, 
and  Computers 

(3)  HQDA  -  Headquarters,  Department  of  the 

Army 

(4)  MACOM  -  Major  Command 


Table  1-2. 


Systems  Requiring  Independent  Test 
and  Evaluation 


Najor  tystOBi 

Indsoandsnt  Evaluation 
fiovt.  Oavalopawital  Oparaticral 

Acquisition 

Category  (ACAT)  I, 

II,  (MSCR) 

Independent  Evaluation 
Report  (lER) 

Operational  Assess* 
ment  (QA) 

Early  QA  (EQA) 

TER 

Class  1.  II,  III,  IV, 
V*  <AIS) 

lER 

OA/EOA/TER 

mai 

ACAT  Ill,  tv  (NSCR) 

Independent  Assessment 
Report  (lAR) 

Abbreviated  Operational 
Assesssient  (AOA) 

Class  V*  (AIS) 

lAR 

AOA 

Class  V*,  VI  (AIS) 

No  retirement  for  independent  evaluation 

(Requires  test  plans,  test  casas/procedures, 
test  reports  and  evaluation  by  the  tester.) 

*  Option  for  independont  cvoluation  (s  dottraintd  by  0tSC4 


Section  IV  Independence  in  Software  Teat  and  Evaluation 
1-16.  Application 

Independence  in  testing  and  reporting  channels  promotes 
objectivity  in  test  and  evaluation  activities.  Army 
requirements  for  independent  testing  are  cited  in  AR 
73-1.  There  are  three  basic  levels  of  independence. 

a.  Independence  within  the  development  organization 
includes  quality  assurance  (QA)  and  test  personnel  who 
report  through  a  different  chain  of  management  than  the 
software  designers  and  coders. 

b.  Independence  within  the  PM  matrix  organizations 
includes  quality,  IV&V,  and  test  personnel  who  provide 
evaluation  or  testing  for  the  PM,  but  report  through 
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subordinate  conunand  (MSC)  or  major  command 
(MACOM)  rather  than  the  PM's  chain  of  command, 
c.  Independent  developmental  and  operational 

Examples  include  personnel 

from  Test  and  Evaluation  Command  (TECOM) ,  Operational 
Evaluation  Command  (OEC) ,  Army  Materiel  Systems 
Analysis  Activity  (AMSAA) ,  Test  and  Experimentation 
Conunand  (TEXCOM) ,  Combat  Systems  Test  Activity  (CSTA) 

Agency  (LEA) ,  Corps  of  Engineers 
(COE) /  Intelligence  and  Security  Command  (INSCOM) , 
Health  Services  Command  (HSC)  and  Information  Systems 
Engineering  Command  (ISEC) .  These  personnel  report 
findings  to  Department  of  the  Army  (DA)  decision- 
authorities. 


Section  V 

Software  versus  System  Testing 
1-17.  General 

a.  Unifying  the  T&E  process  requires  that  the 
software  T&E  mission  include  not  only  software,  but  its 
capability  to  perform  as  an  integral  part  of  the  target 
system  support  of  operational  mission  requirements. 
There  ai.  &  two  objectives  to  testing:  demonstration  of 
performance  and  fault  removal. 

b.  Software  TiE  must  address  system-level 
requirements  which  include,  but  are  not  limited  to, 

interoperability  and  interfaces 
with  other  systems,  supportability,  continuity  of 
operations,  and  user  interfaces.  These  aspects  of  the 
total  system  are  part  of  the  T&E  mission  and  must  be 
tested  and  evaluated  with  the  software  functions. 
Specialized  system  tests  which  are  part  of  the  software 
integration  process  include: 

(£1  Peak  load  test,  e.g.,  all  terminals  active. 

(f)  Storage  testing,  e.g.,  verify  capacity  of 
the  system  to  store  a  specified  amount  of  transaction 
data  on  a  dis)c  or  in  other  files. 

(^) ,  Performance  time  testing,  g. ,  response 
time  for  inquiry  when  system  is  fully  loaded  with 
transaction  data. 

.  Recovery  testing,  e.g.,  load  bac)cup  copy  of 

data  and  resume  processing  after  failure,  without  data 
or  integrity  loss. 

Procedure  testing,  e.g.,  determine  clarity 
of  documentation  on  operation  and  use  of  system  bv 
having  users  do  exactly  what  the  manuals  request. 

u.5s»T-c  testing,  e.g.,  determine  how 

users  will  use  the  system  when  processing  data  or 

preparing  reports  (i.e.,  finding  out  how  users  will 
react  to  the  system  in  ways  not  anticipated  by  the 
analyst(s)).  •* 

incremental  test  strategy  allows  a  variety  of 
test  types  which  are  diverse  enough  to  provide  ^ 
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confidence  in  the  effectiveness  of  the  test  process. 

In  addition,  an  incremental  strategy  provides  a  means 
to  identify  and  correct  failures  earlier  and  more 
efficiently.  Figure  1-2  shows  the  relationship  between 
different  levels  of  requirements  and  corresponding  test 
levels. 


1-18.  Test  levels 

a.  Software  tests.  Lower  levels  of  software  tests 
performed  by  the  software  developers/engineers  are 
structured  to  verify  the  accuracy  of  algorithms  and 
computations,  and  to  make  sure  that  the  portions  of 
coded  software  work  lAW  the  design,  meet  the  expected 
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results,  can  handle  erroneous  inputs  and  have  been 
exercised  with  combinations  of  differing  functions. 
Software  engineers /developers  are  responsible  for 
ensuring  that  user  requirements  are  correctly 

peir  designs  and  that,  when  pieces  of 
software  are  integrated,  they  function  as  required. 

b.  Software/system  tests.  As  the  software 
subsystems  are  integrated,  software  developers/ 
engineers  ensure  that  realistic  stress  and 
interoperability  are  verified  in  tests  at  systems 

^^®  opportunity  to  check 

software  requirements  prior  to  technical  tests  run  by 
the  Government  at  the  system  level.  Software  tests  ^ 

Milestone  III  are  normally 
conducted  in  software  development  facilities  and 
laboratories. 

technical  tests.  Technical  system-level 
tests  look  at  the  capability  of  the  software  to  support 
system  performance.  Government-run  technical  tests, 
called  d®v®lopm®ntal  tests  (DTs)  are  conducted  in  the 
laboratory.  Government  testbeds,  and/or  user 
environments  using  qualified  civilian/soldier 
personnel.  Government  develop"  rntal  testing  is 
structured  to  subject  the  syst  to  stress  levels 
commensurate  with  those  that  t.e  mature  system  will  be 

representative  operating  environments. 
These  tests  may  be  structured  to  estimate  the  outer 
limit  of  the  system's  operational  envelope,  if 
required.  Engineering  requirements,  performance,  and 
user  requirements  are  also  examined  during  DT. 
Government  developmental  testing  is  also  conducted  on 
upgrades  and  fixes  after  Milestone  III. 

d.  Operational  tests.  Operational  testing  is 
syst®m-level  testing  which  focuses  on  effectiveness  and 
suitability.  Characteristically,  operational  tests 
involve  the  intended  user  troop  units  or  organizations, 
and  taJce  place  in  realistic  operational  environments. 
Oprational  tests  may  be  performed  at  any  time  during 
the  life  cycle.  An  initial  operational  test  (lOT)  is 
reguir^d  for  the  Milestone  III  decision  to 
produce/deploy  and  field  a  system.  Other  forms  of 
operational  testing  may  be  used  to  assist  in  refining 
user  requirements,  to  develop  standing  operating 
procedures  (SOPs) ,  to  identify  new  or  different  means 
to  employ  the  system,  etc.  Examples  of  operational 
tests  lAW  AR  73-1  Include  early  user  test  (EUT)  early 
user  experimentation  (EUE) ,  limited  user  test  (LUT) 
force  development  test  (FDT) ,  concept  evaluation  ' 
program  test  (CEPT) ,  or  force  development  experiment 
(FDE) .  The  need  for  these  tests  is  determined  by  the 
user  representatives,  PM,  operational  evaluator,  and 
the  acq  lisition  strategy.  Following  system  deployment, 
operational  tests  are  called  on-site  user  tests  (OSUT) , 
or  follow-on  operational  tests  (FOTs) . 
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Chapter  2 

Terms  and  Definitions 
2-1.  Introduction 

Key  to  implementing  the  software  T&E  process  is  the  use 
of  consistent  and  common  terminology.  This  chapter 
establishes  the  set  of  terms  used  throughout  this  part 
of  DA  Pamphlet  73-1  and  references  similar  terms 
currently  used  within  the  AIS  and  MSCR  environments. 
Acronyms  and  definitions  for  many  of  the  terms  are 
contained  in  the  Glossary. 

2“2.  Terminology  cross-reference 

Entries  under  the  heading  "Current  Terms"  in  the 
following  tables  describe  the  T&E  process  in  this  part 
of  DA  Pamhlet  73-1.  Other  terms  shown  are  for 
reference  only. 

a.  This  part  of  DA  Pamphlet  73-1  uses  a  common  set 
of  terms  to  provide  a  singular  approach  to  describing 
tests  across  the  Apy.  Table  2-1  lists  the  current 
terms  and  those  which  have  been  used  in  the  past. 

b.  Organizational  roles  are  presented  in  Table  ?-2 
to  show  the  relationship  between  terms  currently  in  se 
and  any  deviations  in  terms  used  in  this  part  of  DA 
Pamphlet  73-1.  As  the  Army  moves  T&E  to  the  AR  70 
series  from  the  AR  25  series  of  regulations,  additional 
commonality  may  be  achieved. 

c.  Because  MSCR  and  AIS  developments  chose  to  use  a 
variety  of  formats  for  their  technical  data,  two  sets 
of  software  document  classes  are  shown  throughout  this 
part  of  DA  Pamphlet  73-1.  Table  2-3  shows  MSCR 
software  documents  related  to  DOD-STD-2167A  and  MIL- 
STD-490A,  and  AIS  documents  based  on  DOD-STD-7935A. 

Table  2-3  also  includes  documents  pertaining  to 
Government  DT,  independent  developmental  evaluation, 
operational  test,  and  independent  operational 
evaluation.  Two  of  the  documents  have  expanded  their 
soft  “re  role  as  a  result  of  AR  73-1.  They  are  the 

use  functional  description  (UFD)  and  metrics 
repc.  .  3 . 

d.  Table  2-4  is  a  cross-reference  of  AIS  and  MSCR 
related  to  software  development.  Because  there 

is  much  similarity  between  the  terms,  few  changes  have 
been  made  in  the  Current  Term  column. 

e.  The  categories  and  priorities  presented  in  Table 
2“5  provide  a  standard  means  to  score  software 
deficiencies  during  development  and  test.  These  common 
classifications  form  the  basis  for  controlling  entry 
and  exit  objectives,  promoting  standard  interpretation 
of  test  deficiencies  and  fostering  meaningful  entries 
in  the  Army  Test  Incident  Report  System  (ATIRS) . 

f.  Table  2-6  shows  software  and  system-level  T&E 
events  and  the  organizations  responsible  for  those 
events.  Primary  responsibilities,  events,  and  overall 
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chronology  are  depicted,  other  groups  participate 
throughout  the  life  cycle  to  ensure  T&E  involvement, 
user  involvement  and  sharing  of  data. 

g.  Acronyms  used  in  these  tables  are  discussed  in 
detail  in  later  chapters. 
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AIS  TERN 

NSCR  TERN 

CURRENT  TERN 

Priority 

Emergency 

Priority  1 

Priority  1 

Urgent 

Priority  2 

Priority  2 

Urgent 

Priority  3 

Priority  3 

Routine 

Priority  4 

Priority  4 

Routine 

Priority  5 

Priority  5 

Cateoorv 

Technical  P’-oblem 

Software  Problem 

Software  Problem 

Documentat 

Problem 

Documentation 

Problem 

Documentation 

Problem 

Technical  Problem 
Functional  Problem 

Design  Problem 

Design  Problem 

Note:  Current  Term  priorities  and  categories  are 
defined  in  DOD-STD-2167A,  Appendix  C. 
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Chapter  3 

TtE  as  Part  of  Acquisition  and  Development 
3-1.  Introduction 

These  procedures  present  an  iterative,  structured,  and 
comprehensive  approach  to  software  test  and  evaluation 
throughout  a  system  life  cycle.  Specific  application 
of  these  procedures  should  be  tailored  to  the  technical 
and  management  characteristics  of  each  system 
acquisition  program.  Software  T&E  procedures  are 
determined  by  the  level  of  technical  risk  which  can  be 
allowed  in  the  system  acquisition.  This  chapter 
discusses  general  T&E  strategies  which  must  be 
considered  to  provide  effective  T&E  which  is  responsive 
to  program  needs,  user  requirements,  and  fielding  of 
quality  systems. 

3-2 .  T&E  approach 

Software  T&E  is  accomplished  within  the  context  of  the 
overall  system  development  and  test  program.  Software 
T&E  supports  the  overall  concept  of  total  quality 
management  (TQM)  for  the  system  development.  In 
accordance  with  the  TQM  concept,  all  persons  involved 
in  the  software  development  process  are  responsible  for 
impacting  the  quality  of  the  software  product.  The 
following  general  guidelines  constitute  the  minimum 
requirements  for  a  software  T&E  program. 

a.  Software  is  assessed  for  its  ability  to  support 
the  system  effectiveness  and  suitability.  The  extent 
of  software  T&E  applied  must  be  sufficient  to  provide 
an  acceptable  level  of  risk  that  ensures  system 
requirements  and  mission  objectives  will  not  be 
impaired  by  deficiencies  attributable  to  software. 

b.  Per  AR  73-1,  software  T&E  must  provide  data  to 
support  qualitative  and  quantitative  software  metrics. 
These  metrics  serve  as  measures  and  indicators  of  the 
critical  technical  and  operational  characteristics  that 
both  the  software  and  the  integrated  system  need  to 
achieve. 

c.  The  overall  software  T&E  program  must  reflect  a 
systematic  and  measurable  process  in  which  continuous 
software  evaluations  present  a  realistic  and  iterative 
status  report.  Clearly  defined  entry  and  exit 
objectives  for  each  phase,  metrics  and  continuous 
evaluations  are  the  basis  for  a  logical  progression  of 
software  T&E.  This  progression  is  based  on 
demonstrating  achievement  of  objectives  at  each  phase. 
Figure  3-1  portrays  the  general  correspondence  between 
system  life  cycle  phases  and  software  T&E  phases  and 
activities.  The  exit  objectives  for  each  phase  are 
summarized. 

d.  The  software  T&E  program  must  support  the 
progressive  approach  by  effectively  sharing  test  and 
evaluation  results  across  the  life  cycle  phases.  Each 
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element  of  the  software  T&E  program  must  provide  data 
to  support  software  and  system  acquisition  decisions. 

e.  Software  T&E  must  support  the  acquisition 
management  process.  Individual  programs  and 
acquisition  strategies  determine  the  scope  of  software 
T&E  programs  to  fit  the  acquisition. 

3-3.  Test  and  evaluation  considerations 

3-  The  most  efficient  and  effective  use  of 
resources  is  the  sharing  of  data  among  developers, 
testers  and  evaluators.  For  any  system,  software 
testers  and  evaluators  need  to  approach  and  plan  their 
strategies  with  an  understanding  of  the  complexity  of 
the  system,  the  development  cycle,  the  acquisition 
category,  the  deployment  philosophy,  and  the  other 
players  involved.  Factors  such  as  the  acquisition 
category  determine  the  reporting  and  approval 
requirements,  and  the  requirement  for  independent 
evaluation.  Materiel  release  requirements  are  an 
additional  consideration  since  the  PDSS  activity  has  to 
provide  a  software  supportability  and  suitability 
statement  for  fielding  decisions. 

b.  Differences  in  the  inherent  characteristics  of 
system  computer  resources  and  software  architectures 
dictate  a  variety  of  software  T&E  methods  and 
procedures.  These  methods  and  procedures  are  required 
to  address  different  types  of  software  structures  and 
to  collect  different  types  of  data  which  are  meaningful 
for  software  evaluation.  The  most  significant 
requirement  for  selecting  software  test  methods  is 
understanding  the  objectives  and  data  requirements  for 
that  phase  of  software  T&E.  The  amount  and  intensity 
of  software  testing  must  be  established  early  in  the 
development  process.  Test  procedures  must  be  developed 
simultaneously  with  the  software.  Selection  of 
software  T&E  techniques  depends  on  several  factors, 
including: 

(1)  Overall  complexity  of  the  system  and 
software. 

(2)  Extent  of  critical  technical  and  operational 
characteristics . 

(3)  Level  of  technical  risk  in  the  software's 
development. 

(4)  Target  environment  and  intended  end  use. 

c.  Single-site  systems  are  systems  which  reside 
only  at  one  location.  As  part  of  the  strategy  for 
these  systems,  additional  T&E  considerations  include 
arrangements  for  testing  and  realistic  methods  of 
testing.  An  additional  test  design  consideration  is 
operational  tests  during  PDSS  as  there  may  be  test 
limitations  associated  with  single-site  systems.  This 
may  dictate  the  use  of  combined  DT/OT.  An  example  of  a 
single-site  system  is  the  Army  central-site  financial 
systems  which  reside  on  large  mainframes  at  one  location. 
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(1)  Test  designs  for  single-site  systems  must 
account  for: 

(a)  Loading  and  running  the  system  in 
specialized  test  regions  which  are  partitioned  from  the 
production  regions. 

(•b)  Running  the  volumes  and  stress  loading 
necessary  to  simulate  or  stimulate  expected 
interactions  with  users,  other  systems  or  the  external 
environment.  This  may  require  the  use  of  models. 
(Models  for  T&E  purposes  are  required  to  be  accredited 
by  an  activity  independent  of  the  model  developer.) 

(c)  Ensuring  that  all  cycles  are  executed, 
whether  by  models  or  actual  testing,  including  end-of- 
year,  and  other  cycle  roll-ups. 

(2)  Test  and  evaluation  strategies  for  these 
systems  must  consider  all  opportunities  to  gather  data 
throughout  design  and  development.  Models  developed 
for  stress  loading  systems  for  T&E  purposes  must  be 
accredited  prior  to  use  by  independent  evaluators. 
System  contractor  models  will  not  be  used  for 
operational  testing  (OT)  and  evaluation  without  an 
independent  certification  that  they  accurately  portray 
the  external  interactions. 

d.  Single-user  systems  are  systems  designed  for  use 
by  one  user  organization.  Presence  of  a  single-user 
organization  allows  direct  user  involvement  throughout 
development  and  the  T&E  process.  It  is  essential  that 
the  operational  users  be  part  of  the  T&E  planning  to 
ensure  that  specialized  operations  are  not  overlooked. 

e.  Non-theater/tactical  systems  include  sustaining- 
base  and  strategic  systems.  Strategic  systems  normally 
are  managed  under  DOD  and  JCS  publications  and, 
occasionally,  AR  525-1.  Strategic  systems  may  be 
developed  under  25-series  policies.  Sustaining-base 
systems  provide  the  capability  to  raise,  organize, 
train,  equip,  deploy  and  sustain  Army  forces  in 
accomplishing  the  operational  mission.  These  systems 
are  loc*  1  at  the  garrison  or  installation  level  and 
do  not  t  jSically  move  into  the  battlefield  during 
mobilization  or  wartime.  They  may  perform  differing 
missions  during  peacetime,  mobilization,  wartime  and 
demobilization,  and  they  may  reside  on  mainframes  or  be 
centrally-sited  single-user  systems.  These  large  scale 
systems  have  unique  T&E  considerations.  (Refer  to  AR 
25-3. ) 

f.  Theater/tactical  systems  are  systems  which 
operate  in  a  defined  area  of  the  operational  theater. 
These  are  systems  which,  unlike  sustaining-base,  focus 
on  combat  and  wartime  missions.  Theater/tactical 
systems  must  respond  to  additional  design 
considerations  related  to  operational  environment  and 
real-time  processing.  (Refer  to  AR  25-3.) 

g.  MSCR  refers  to  computer  resources  acquired  for 
use  as  integral  parts  of  weapons  systems,  command  and 
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control,  communications,  intelligence  and  other 
tactical  or  strategic  systems  and  their  support 
systems.  This  term  applies  to  all  computer  resources 
(hardware,  software,  documentation,  data,  etc.) 
associated  with  specific  program  technical  T&E,  user 
TiE,  and  ppst-deployment  support  including  trainers, 
automatic  test  equipment,  land-based  test  sites,  and 
system  integration  and  test  environments.  Software  T&E 
must  address  several  unique  characteristics  including 

fault  tolerance  requirements. 

(Refer  to  AR  70-l.) 

h.  Automated  information  systems  encompass  the 
functions,  resources,  equipment,  software  and 
activities  associated  with  one  or  more  of  the 
disciplines  of  automation,  telecommunications,  visual 
information,  publications  and  printing,  and  records 
management.  They  are  most  often  non-theater/tactical 
systems.  (Refer  to  AR  25-3.) 


3-4.  Security  accreditation  and  its  relation  to 
software  T&E 

a.  All  AIS  and  MSCR  systems  which  contain 
information  require  safeguards  to  protect  against 
compromise,  subversion,  or  unauthorized  manipulation. 
Information  is  safeguarded  by  hardware  security 
software  security,  procedural  security,  communications 
security,  personnel  security,  physical  security, 
networJc  security,  electronic  security,  and  control  of 
compromising  emanations.  Security  accreditation  is  the 
process  which  reviews  the  layers  of  safeguards. 
Accreditation  is  the  procedure  for  determining  the 
level  of  security,  the  risks,  vulnerabilities,  and  the 
planned  safeguards. 

b.  Accreditation  results  from  the  process  of 
collecting,  analyzing,  and  submitting  information 
pertaining  to  the  security  approval  for  software 
systems.  The  process  of  security  accreditation  is 
described  in  Army  Regulation  380-19  for  "Information 
Systems  Security."  This  regulation  introduces 
information  system  security  (ISS)  as  a  discipline 
encompassing  communication  security  (COMSEC) ,  comouter 
security  (COMPUSEC) ,  electronic  security  (ELSEC) ,  and 
control  of  compromising  emanations  (TEMPEST) .  The 
designated  accreditation  authority  (DAA)  is  a  senior 
management  official  who  has  the  authority  and 
responsibility  to  decide  to  accept  or  reject  the 
security  safeguards  prescribed  for  an  automated 
information  processing  system  and  who  may  be 
responsible  for  issuing  an  accreditation  statement  or 
certificate  that  records  the  decision  to  accept  those 
safeguards  for  his/her  department,  agency,  or  service 

A  DAA  will  be  identified  for  all  AIS%nd  MSCR 
processing  classified  or  unclassified-sensitive 
information.  (Reference  Section  3-8  of  AR  380-19  for 
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the  appointment  of  a  DAA  which  is  dependent  upon  the 
sensitivity  or  classification  of  the  system.) 

c.  Software  test  and  evaluation  personnel  assist  in 
accreditation  efforts.  AR  380-19  states  that  software 
design  and  test  activities  are  part  of  the  software 
certificat_ion  and  accreditation  process.  T&E 
activities  and  responsibilities  for  accreditation  are: 

(1)  Audits  of  control  procedures  used  to  develop 
software. 

(2)  Requirements  for  software  to  be  completely 
tested  before  becoming  operational. 

(3)  Requirements  for  testing  which  must  include 
valid  and  invalid  data. 

Software  testing  is  not  complete  until  all  security 
requirements  have  been  examined  and  expected  results 
are  attained. 

d.  The  accreditation  process  requires  the  following 
(see  AR  380-19  for  details) : 

(1)  Development  and  modification  of  a  security 
plan  (lAW  AR  380-19,  Appendix  C) . 

(2)  Determination  of  accreditation  objectives. 

(3)  Completion  of  risk  marsgement  reviews. 

(A)  Definition  of  propose  operations. 

(5)  Selection  of  security  , ountermeasures. 

(6)  Certification  (lAW  AR  380-19). 

(7)  Development  of  a  security  guide. 

e.  Software  T&E  personnel  need  to  be  familiar  with 
the  safeguards  for  security  and  the  accreditation 
requirements  lAW  AR  380-19.  Security  issues  and 
requirements  are  an  integral  part  of  the  comprehensive 
T&E  strategy  for  fielding  secure,  trusted  systems. 

3-5.  Metrics  and  T&E 

a.  Metrics  are  technical  and  management  tools  used 
to  highlight  and  identify  potential  problems  and/or 
deficiencies.  They  are  quantitative  and  qualitative 
measures  and  indicators  which  help  focus  management 
attention  and,  if  appropriate,  resources  on  the 
prevention  or  correction  of  problems. 

b.  Metrics  measure  and  provide  feedback,  impacting 
both  the  product  and  process,  and  enable  managers  to 
continuously  improve  the  process.  Metrics  are  an 
integral  aspect  of  controlling  and  reporting  software 
T&E  activities.  They  are  required  for  Army  software 
development  and  T&E.  Metrics  may  be  developed, 
collected,  and  used  by  many  people  (e.g.,  developers, 
evaluators,  PDSS,  software  engineers,  testers,  SQA, 

IV&V,  etc.).  Metrics  are  reported  to  evaluators, 

PM/PEO,  and  review  council  decision-makers.  Figure  3-2 
depicts  representative  metrics  during  the  life  cycle  of 
a  typical  waterfall  model  acquisition  strategy.  The 
definitions  and  uses  of  these  metrics  are  described  in 
Software  T&E  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 
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and  devices  which  may  constitute  a  test  bed 
environment.  Developers,  software  engineers,  testers 
and  evaluators  should  identify  and  plan  for  these  needs 
early  in  the  system  life  cycle  to  ensure  availability 
at  test  time. 

3-7.  Prototypes  and  TtE 

Software  prototypes  are  normally  development  tools,  not 
a  testing  technique  or  a  substitute  for  system 
development,  testing  and  configuration  management  (CM) . 
However,  these  tools  can  be  an  important  part  of  the 
software  development  methodology.  Informal  software 
releases  refer  to  software  which  is  used  for 
engineering  design  and  development.  These  informal 
software  products  are  usually  poorly  documented,  lacJc 
adequate  design  certification,  and  are  not  subject  to 
configuration  management  procedures.  Informal  releases 
will  not  become  part  of  an  approved  releasable  software 
baseline  because  they  present  an  unacceptable  level  of 
technical  risk  until  they  are  part  of  the  approved 
baseline.  Informal  releases  of  prototypes  are 
encouraged  to  be  utilized  for  user  demonstrations, 
early  looks  at  the  software-human  interface  designs, 
and  early  requirements  refinements.  Formal  T&E  demands 
defined  and  documented  requirements;  informal  T&E  does 
not  necessarily  require  these.  In  the  event  that  the 
development  plan  allows  for  the  eventual  operational 
use  of  prototype  software,  those  prototype  units  must 
be  properly  tested  and  documented. 

3-8.  T&E  in  support  of  tailored  accfuisition  approaches 
The  Development  and  Acquisition  Test  and  Evaluation 
Procedures,  of  this  document  describes  the  software  T&E 
process  for  the  standard  acquisition  strategy. 

However,  tailored  acquisition  approaches  are  used  and 
need  to  be  considered  for  their  risks  and  impact  on  the 
software  T&E  process.  Three  examples  of  tailored 
acquisition  approaches  are:  non-developmental  items, 
the  use  of  concurrent  or  combined  DT  and  OT,  and 
accelerated  development/ fielding. 

a.  Non-developmental  items  (NDI)  and  software. 

(1)  The  NDI  approach  is  the  acquisition  of  a 
product  that  is  already  developed  for  another  purpose 
or  another  user •  It  is  chosen  because  it  is  supposed 
to  be  capable  of  performing  the  mission.  NDI  systems 
can  be  the  means  to  field  a  system  faster  and  cheaper 
with  little  or  no  hew  development  effort.  For  many 
systems,  computer  hardware  is  NDI. 

(2)  Software  frequently  fits  in  the  NDI  category 
because  it  may  be  off-the-shelf.  However,  NDI  software 
may  not  always  be  off-the-shelf,  but  may  be  a 
combination  of  off-the-shelf  and  newly  developed 
software.  NDI  software  which  requires  modification  or 
system  integration  must  undergo  software  T&E.  If  the 
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developer  is  required  to  perform  these  tears,  the 
Government  T&E  community  reviews  the  test  plans  and 
monitors  the  performance  to  evaluate  the  adequacy  of 
testing.  Results  of  this  evaluation  are  reported 
during  readiness  reviews  prior  to  the  start  of 
Government-  testing. 

(3)  NDI  software  may  not  require  Government 
developmental ing.  In  all  cases,  operational  testing  is 
conducted  to  ensure  that  the  NDI  software  can  perform 
the  mission.  NDI  operational  testing  is  performed  by  a 
designated  Government  organization.  The  requirement 
for  independent  evaluation  of  NDI  software/systems  will 
be  determined  lAW  AR  73-1. 

(4)  NDI  executive  software  will  be  validated 
during  operational  testing  using  a  representative 
functional  application.  No  executive  software  can  be 
released  or  used  with  an  application  until  it  has 
successfully  completed  a  operational  test.  (Executive 
software  includes  compilers,  utilities,  operating 
systems,  special  customized  system  software,  etc.) 

(5)  NDI  software  testing  emphasizes  the 
following: 

(a)  That  the  software  meets  contract 
requirements. 

(b)  Developer's  enhancements  or  adaptations 
should  perform  correctly  and  not  introduce  new  defects. 

(c)  Developer-provided  documentation  is 
adequate  to  permit  suitable  operation  and  maintenance 
of  the  software  product  in  its  intended  user  and 
maintenance  environment. 

(d)  Software  meets  the  user's  stated 
requirements  and  the  operational  mission. 

(6)  Software  developments  which  use  NDI  hardware 
must  also  plan  to  ensure  that  the  software  can  run 
effectively  and  efficiently.  Special  testing 
considerations  include: 

(a)  Software  integration  with  hardware  should 
demonstrate  that  performance  characteristics  are  met. 

(b)  Th  ntegrated  system  (hardware, 
software,  documentation)  meets  user's  stated 
requirements,  and  is  mission  effective  and  suitable. 

(c)  Hardware  PM,  software  PM,  and  software 
developers  must  work  closely  to  ensure  that  the  system 
works.  Early  availability  of  target  hardware  during 
development  reduces  risks  associated  with  integration. 

(7)  The  T&E  community  needs  to  carefully  review 
and  understand  plans  which  incorporate  NDI  in  the 
acquisition  strategy.  Documentation,  evidence  of 
testing,  extent  of  software  modifications  and  eventual 
software  supportability  are  areas  which  impact  T&E. 

b.  Concurrent  and  combined  DT  and  OT. 

(1)  Concurrent  DT  and  OT  occurs  when  DT  and  OT 
are  conducted  on  separate  identical  systems 
simultaneously.  Combined  DT  and  OT  are  conducted  on 
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the  same  system  with  the  DT  portion  usually  preceding 
the  OT  portion.  In  some  cases,  combined  DT  and  OT  are 
conducted  simultaneously  on  the  same  hardware  platform. 
For  any  of  these  approaches,  separate  evaluations  are 
performed  by  developmfental  and  operational  evaluators, 
if  required.  All  these  approaches  have  inherent  risks, 
although  a  combined  sequential  approach  is  the  least 
risky  of  the  three  because  technical  aspects  are  dealt 
with  first.  Since  completion  of  a  successful  DT 
usually  serves  as  a  basis  for  entering  OT,  the  lack  of 
DT  forces  developer  testing  to  be  the  basis  of  OT. 

Thus,  the  benefits  of  wringing  out  software  problems 
affecting  system  performance,  as  done  in  DT,  are  lost. 
The  risk  of  failing  OT  is  increased. 

(2)  Risks  of  combined  or  concurrent  DT  and  OT 
can  be  reduced  by  technical  and  operational  evaluators 
having  early  involvement  in  defining  test  strategy. 

Risk  factors  such  as  whether  existing  or  new 
technologies  are  used  and  lessons  learned  from  past 
failures  and  successes  of  similar  system  acquisitions 
should  be  considered.  This  will  lead  to  the  most 
ffislistic  and,  in  the  long-term  cost  effective, 
approach  to  successful  .system  acquisition. 

c.  Accelerated  development/fielding.  This  strategy 
is  a  variant  of  incremental  acquisition  and  is 
applicable  for  software  intensive  systems,  particularly 
AIS,  where  user  functionality  can  be  cleanly  divided 
into  standalone  blocks  which  can  be  designed, 
developed,  and  tested  at  an  operational  level  with  a 
high  degree  of  independence  and  hopefully  parallelism 
as  well.  Each  successive  block  provides  increased 
functionality  for  the  users  or  adds  integration 
capability  necessary  for  communicating  with  other 
systems.  An  lOT  for  each  software  block  is  performed 
on  a  testbed  culminating  in  an  lOT.C  and  Milestone 
III.C,  where  "C"  refers  to  fielding  certification.  The 
testbed  is  expanded  as  needed  for  each  successive  lOT. 
Following  a  successful  lOT.C,  authorization  for 
purchase  of  the  full  hardware  and  software  complement 
for  all  users  occurs  and  the  system  is  fielded  for 
their  employment.  This  strategy  is  primarily 
appropriate  for  low  to  medium  risk  programs  when  user 
requirements  can  be  fully  defined  and  very  little 
special  purpose  hardware  or  development  software  is 
required.  This  strategy  makes  heavy  use  of  NDI 
hardware  and  software  and  prototyping. 

3-9.  TtE  and  risk  management. 

As  well  as  describing  current  Army  testing  policies, 
the  procedures  and  guidelines  in  this  part  of  DA 
Pamphlet  73—1  attempt  to  show  the  relationship  of 
software  products  and  functions  as  integral  components 
of  the  larger  systems  of  which  they  are  a  part. 

Software  TiE,  continuous  evaluation,  incremental 
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each  of  the  phases,  in  the  order  in  which  they  are 
normally  achieved  in  a  traditional  system  acquisition. 
For  each  item,  both  the  MSCR  and  AIS  terms  are 
provided,  and  the  configuration  management  efforts  are 
summarized. 

(1)  -Concept  exploration/definition  or  concept 
development  (Milestone  0  through  I,  may  also  be  through 
Milestone  II  for  MSCR) .  The  configuration  management 
function  supports  the  initial  test  and  evaluation 
efforts  by  identifying  and  controlling  the  approved 
versions  of  the  initial  planning  documents,  such  as  the 
acquisition  strategy,  the  system  decision  paper  and  the 
test  and  evaluation  master  plan  (TEMP) .  Normally  the 
system/segment  specification  and  the  functional 
description  are  authenticated  toward  the  end  of  this 
phase,  establishing  the  MATDEV's  functional  design 
baseline.  A  configuration  control  board  is  established 
by  the  Army  Program/ Project  Office  and  formal 
configuration  control  is  initiated.  Any  software 
developed  for  exploratory  reasons  during  this  phase  is 
not  normally  subject  to  configuration  management, 
although  the  specification  and  control  of  the 
functional  requireme  ts  for  this  software  may  be 
desirable.  The  conf.  duration  management  office 
develops  the  system  configuration  management  plaff^t^P) 
and  prepares  the  configuration  management  related 
inputs  for  the  concept  demonstration  or  validation 
contracts. 

(2)  Requirements  definition  and  software  design 
(AIS  Milestones  I  -  II,  MSCR  Milestones  II  -  III).  The 
configuration  management  function  supports  the  test  and 
evaluation  efforts  by  identifying  and  controlling  the 
system  baselines  against  which  the  system  is  to  be 
tested  and  evaluated.  During  this  phase,  system 
engineering  efforts  and  analyses  are  conducted  to 
determine  the  best  allocation  of  system  requirementrs 
among  hardware,  software,  personnel,  and  facilities. 

The  software  requirements  are  partitioned  into  'a  s=^t  of 
software  requirements  specifications  (SRS) ,  interf  e 
requirements  specifications  (IRS),  or  system/ subs  rcem 
specifications  (SS) .  One  or  more  software  specifi¬ 
cation  reviews  (SSRs)  are  held  to  determine  the 
adequacy  of  the  software  requirements  allocation 
efforts  and  the  adequacy  of  the  software 
specifications.  The  set  of  development  specifications 
may  be  authenticated  toward  the  end  of  this  phase, 
resulting  in  the  establishment  of  the  allocated 
baseline,  or  deferred  to  the  next  phase. 

p)  Software  coding  through  system  integration 
(AIS  Milestone  II  -  III,  MSCR  Milestone  II  -  III).  The 
configuration  management  function  supports  the  test  and 
evaluation  efforts  by  identifying  and  controlling  the 
contractor's  test  planning  documents  such  as  the 
software  development  plan  (SDP) ,  software  test  plan 
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(STP) ,  or  the  test  plan  (PT) .  In  this  phase,  the 
software  elements  of  the  system  are  developed,  tested 
and  evaluated.  As  the  contractor's  test  program  is 
defined  and  implemented,  the  configuration  management 
function  stores  and  controls  the  test  documentation, 
such  as  the  software  test  descriptions  (STDs).  The 
configuration  management  function  also  controls  and 
stores  the  products  of  the  software  development,  as 
they  are  being  coded,  tested  and  evaluated,  such  as  the 
software  design  documents  (SDDs) ,  interface  design 
documents  (IDDs)  or  software  unit  specifications  (US) 
and  data  base  specifications  (DS) .  A  functional 
configuration  audit  (FCA)  is  accomplished  for  each 
software  item  to  ensure  that  the  item,  as  tested  and 
evaluated,  meets  its  requirements  as  identified  in  the 
allocated  baseline.  The  FCA  uses  the  software  test 
report  (STR)  or  the  test  analysis  report  (RT)  as  one  of 
the  bases  for  evaluation  of  the  software  performance. 

A  physical  configuration  audit  (PCA)  is  normally 
conducted  on  the  first  production  configuration  item 
(Cl),  which  may  occur  during  the  low  rate  initial 
production  or  after  the  Milestone  III  decision  on  the 
first  full  rate  production  item.  The  PCA  is  conducted 
to  ensure  that  the  coded  software  in  the  production 
item  is  accurately  and  completely  reflected  in  its 
associated  documentation.  After  successful  completion 
of  the  FCA  and  PCA,  the  Cl's  configuration 
identification  is  entered  into  the  product  baseline. 

(4)  Production/deployment  or  deployment  and 
operations  (AIS  and  MSCR  Milestones  III  -  IV) .  Since 
the  software  is  normally  fully  developed  during  the 
engineering  and  manufacturing  development  or  system 
development  phase,  the  software  activities  are 
concerned  with  resolving  identified  deficiencies, 
making  changes  to  the  software  that  are  required  by 
production  changes  in  the  hardware,  and  identifying  and 
developing  enhancements  to  the  software.  The 
transition  of  software  support  from  the  contractor  to 
the  Life  Cycle  Software  Engineering  Center  (LCSEC)  or 
the  Central  Design  Activity  (CDA)  is  normally  executed 
in  this  phase,  depending  on  the  suitability  of  the 
software  and  the  ability  of  the  LCSEC/CDA  to  perform 
adequate  life  cycle  software  support.  The 
configuration  management  function  supports  the  test  and 
evaluation  efforts  by  identifying  and  controlling  the 
approved  versions  of  transition  and  installation 
documents,  such  as  the  software  support  transition  plan 
(SSTP)  or  implementation  plan  (IP) .  The  major 
configuration  management  effort  during  this  phase  is 
concerned  with  maintaining  change  control  and  status 
account  of  the  established  baselines.  The  information 
in  the  technical  data  package,  including  the  software 
documentation  programmer's  manuals,  version  description 
document  (VDD) ,  etc.,  is  entered  into  the  Army's 
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Automated  Technical  Data/Conf iguration  Management 
System  (TD/CMS) .  All  changes  to  the  established 
baselines  must  be  formally  processed,  tested  and 
approved  prior  to  implementation. 
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Chapter  4  Building  the  Software  TSE  Team 
4-1.  Purpose 

This  chapter  discusses  forming  the  PM's  T&E  team  and 
the  various  T&E  working  groups.  The  acquisition 
planning  performed  by  the  PM  also  requires  input  from 
software  T&E  organizations  to  ensure  that  a  viable  T&E 
program  is  established  and  maintained.  The  types  of 
personnel  identified  for  the  team  are  independent 
testers  (developmental  and  operational),  independent 
evaluators  (developmental  and  operational) ,  SQA 

cycle  software  engineering  personnel, 

applicable.  Early  meetings  with 
the  PM,  PEO,  materiel  developers,  (or  contractors,  if 
applicable)  and  the  other  members  of  the  team  are 
paramount.  These  meetings  address  the . operational  and 
Government  developmental  test  concepts.  Issues 
discussed  are:  test  bed  design,  development  strategy, 
configuration  management,  instrumentation,  test  files 
proposed  test  dates  and  the  T&E  concept.  ' 


4-2.  Software  T&E  organizations  and  responsibilities. 

a.  The  organizations  responsible  for  the  following 
activities  are  identified  by  the  PM.  Many  of  the 
organizations  must  be  involved  during  mission  need 
determination  in  order  to  provide  early  planning 
assessment,  and  structure  to  the  system/software ^ 
development  and  T&E.  The  software  T&E  personnel  are 
part  of  the  PM|s  system  acquisition  team  to  provide  T&E 
support  which  is  technically  knowledgeable  in  software 
design,  performance,  and  capabilities.  Their  purpose 
IS  to  enhance  and  improve  the  exchange  of  information, 
prevent  duplication  of  testing  and  data  collection,  and 
provide  the  PM  with  information  to  make  decisions. 

Table  4-1  Identifies  the  T&E  personnel  and  their 

describes  the  general  categories 
of  T&E  team  players  and  provides  examples  to  identify 
the  Army  organizations.  Figure  4-1  depicts  T&E  team 
level  of  involvement  throughout  the  T&E  process.  This 
represents  a  normal  acquisition.  Members  of  the  team 
are: 


(1)  Users'  representatives.  These  are  the 
designated  combat  developers,  functional  proponents 
and/or  proponent  agents.  They  define  and  refine  user 
needs  and  requirements  to  the  developer,  T&E  people  PM 
and  others.  They  are  integral  to  ensuring  that  the^ 
system  meets  the  stated  user  needs  and  requirements. 
They  must  be  involved  throughout  the  entire  process  as 

a  constant  monitor  to  achieve  the  user  needs. 

(2)  Software  development  organization.  The 
software  developer  is  the  organization  designated  to 
design  and  develop  the  software.  A  software  developer 
will  be  identified  when  the  concept  evaluation  requires 
advanced  demonstration  of  software  as  part  of  the 
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concept  definition  process  and  to  reduce  risk 

software  developer(s)  used  during  nrirph««  r.. 

may  not  be  the  same  as  those  who  will  desian  anrf  ^  ^ 

develop  the  production  software.  The  software 

developer  is  responsible  for  testing  the  software 

during  development  as  stated  in  work  directives  or  thP 

n  software  developerl  a^^ 

(CDA) ,  contractors,  center  for 

»"■»  life  Cycle  SoJtwIre 
Engineering  Centers  (LCSEC) .  soroware 

(3)  Software  developer's  internal  OA  Thie; 

SsfJifS  “®  softw«e  de^lopei 

the  internal  QA  for  software 
design  development  and  some  test  activities  Theco 

An  exa^p?e  orai^Ar^y  s^lJarf devefopl?"f ^-i^’na^OA' 

organization  is  the  Quality  Improvement  Office  (QIO)^ 

PM.c  quality  assurance  organization.  ’ 

PM  s  SQA  personnel  are  generally  matrixed  to  the  pm 

evaluatio^J®?h"®'''’i®  providing  software  quality' 

hv  Jhf  iS  0^?'"°“^'’°“*.’"'®  acquisition.  As  ?equesLd 

rLpsik .  :o?tea^?r:p^2ii?c"f 

?e^1iwr!c^R?**ils"t’^?^L*Ineis"“?SiieSI^^^^J  ’ 

®f forts  also  include  management  or 
Lppo?i^?ransitioJ^®  Program  and  software 

is  the  Quality  Assurance®S??ec?o«te 

oroaniiation^n’^hE^?^"*®”^  support  organization.  This 
rganization  is  the  Army  agency  responsible  for 

software  maintenance  management.  This  oroani ^ar-i 

suSy 

efforts  nay  include  presiding  over  softw«^re^aXd 

oT^a«^^!p!unrL*'tnT^;fL^"?4  L1t"^>g?a"^ 

««?aojo?;.  ®^®  "“®- 

groups  ‘provL"f the 

software/system  evaluators.  Independent  telJe^rLe 

not  part  of  the  development  organization  ?heJ 

the  PM  with  independent  developmental  testing  or^  ^ 
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operational  testing  support  throughout  the  life  cycle. 
Operational  testers  are  required  to  be  separate  and 
independent  from  the  materiel  developer  (MATDEV) ,  and 
the  procuring  and  using  agencies.  Examples  are  OPTEC- 
TEXCOM  (operational  tester),  TECOM  (developmental 
tester) ,  and  ISEC-QAD  (developmental  tester) . 

(7)  Independent  evaluation  organizations.  For 
those  systems  recjuiring  independent  evaluation,  they 
participate  in  concept  definition  evaluations  by 
providing  early  developmental  or  operational  system/ 
software  evaluations.  These  evaluations  are  based  upon 
data  available  during  this  part  of  the  life  cycle. 
Issues  and  criteria  determine  what  data  should  be 
evaluated.  Government  developmental  and  operational 
evaluators  are  not  part  of  the  development  organization 
or  the  PM's  office.  Developmental  and  operational 
evaluators  will  report  to  the  decision-makers  at  each 
milestone.  For  systems  which  have  no  IV4V  agent, 
developmental  and  operational  evaluators  may  be 
required  to  prepare  additional  reports  for  the  CDR. 
Operational  evaluators  must  be  separate  from  the 
materiel  developer  and  the  procuring  and  using 
agencies.  Examples  are  ISEC-SAO  and  OPTEC-OEC. 

(8)  Independent  verification  and  validation 
organizations.  The  PM  determines  the  level  of  software 
risk  associated  with  the  program.  The  PM  may  elect 
(and  is  strongly  encouraged)  to  use  IV4V  personnel  as  a 
means  to  reduce  and  manage  risk,  provide  additional 
software  design  evaluation,  witness  developer  tests, 
and  perform  other  activities.  Independent  means 
independent  of  the  software  development  organization. 
The  earlier  the  IV&V  personnel  are  involved,  the  better 
the  structure  of  software  T&E.  For  complex  and 
sophisticated  systems,  the  developmental  and 
operational  evaluators  use  the  inputs  provided  by  IV&V 
agencies  to  ensure  that  software  is  sufficiently 
mature.  An  IV&V  agent  may  be  either  a  government  or 
contractor  organization. 

These  organizations  are  identified  and  assigned 
early  so  that  T&E  planning  is  a  smooth  process  that 
does  not  disrupt  the  program.  It  also  gets  the  T&E 
personnel  integrated  into  the  project  as  part  of  the 
PM's  program  and  acquisition  team.  This  enables 
software  T&E  personnel  to  determine  the  extent  to  which 
data  can  be  shared,  and  tests  and  evaluations  combined. 
The  software  T&E  team  can  better  support  the  PM  by; 

(1)  Sharing  evaluations. 

(2)  Identifying  deficiencies  early. 

(3)  Providing  alternative  T&E  strategies. 

(4)  Keeping  the  PM  and  decision-makers  informed. 

4-3.  Working  groups 

Software  T&E  teamwork  is  enhanced  through  the  effective 
use  of  working  groups.  These  working  groups  are 
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established  by  the  PM  who  is  responsible  for  their 
Products.  During  mission  need 

phasic  exploration/definition 

??cSr;Tiwi,™  tv/nToti 

computer  resources  working  group  (CRlG^^il  ^ 

established;  These  worKiIg%roSpi“re'dU%SsL“d  below. 

team  ^It  primary  test  and  evaluation 

effSrt  T^I  total  T&E 

program  for  the 

t-eer  software,  and  integrates  the  varvina 

test,  evaluation,  and  data  requirLents. 

»es*u  ^  .  ®  Structure  of  the  TIWG  and  its  orincinai 

Jof?wa«®Tsrsuppo?r 

TIWO  to  pertot/a!r^:2  d^St^es  ?:J%^riJs%"e\“*'  " 

T5M  •  ”“®  members  of  the  TIWG  are  the» 

oS;ra??oS^n'"^  evaluators  (developpental  and 
ooJJatio^ai  '  ,^"^®P®"'^®"t  testers  (developmental  and 

^sP^esentatives,  logisticians  oost- 
deployment  support  personnel,  anr  trainers.  Associ!?r 
members  include  the  software  quaj'ty  assurance 

JngineJ??^^^''®'  '^®''®^°P“®"t  tester,  and  system 

(4)  Duties  performed  by  the  TIWG  are: 

cofti-H  insi -fu  “  Prepares  a  charter  and 

coordinates  with  principal  members.  The  charter 

membership,  duties,  author!?',  pJw^Lres 

for  meffling  disputes,  and  objectives  of  the  TIWG. 

inniir  an,^  ^  Charter  other  groups  to  provide 

input  and  support  to  the  TIWG.  For  example  if  the 
software/system  is  used  by  other  services  a  ^ 

TIWG^''^Fo?^ar"7S^?""^"^  provides  input  to  the  ^ 

TIWG.  For  AR  70-1  systems,  the  CRWG  provides  the 

software  T&E  input  to  the  TIWG  and  informs  the  TIWG  nf 
the^software  HE  status  throughout  the  ???t??'l!!r 

.  __  TIWG  prepares  the  TEMP  for  the  pm  tho 

reeLnf-KM^r  members 

responsibility  for  the  parts  of  the  TEMP.  Parts  i  and 

II  are  prepared  by  the  pm  with  role  and  resp???fbili?? 
inputs  from  the  principal  members.  Part  III  is  ^ 

prepared  by  the  developmental  tester  and  the 
independent  developmental  evaluator  and  may  include 
input  from  SQA,  ivtv  agent,  and  the  soft??«  S?i???per 
rffJ  prepared  by  the  independent  operational  ^ 

V  independent  operational  evaluator.  Part 

requires  input  from  all  TIWG  members  to  determine  t&e 
resources  (e.g.,  automated  testing  drivers,  ^ 

simulations  models  used,  users  to  be  involved  in 

testing,  etc.).  The  TEMP  format  is  provided  in  Part 
Two  of  DA  Pamphlet  73-1.  Piroviaea  in  Part 
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(d)  TIWG  handles  all  disputes,  assists  in 
allocation  of  T&E  resources,  detennines  where  data  can 
be  collected  to  answer  issues  and  criteria,  and 
discusses  evaluation  and  test  events. 

(5)  It  is  the  PM's  responsibility  to  charter  the 
TIWG.  It  -is  the  responsibility  of  the  principal 
members  to  ensure  that  it  is  an  effective  group, 
b.  CRWG. 

(1)  The  CRWG  is  a  working  group  used  by  AR  70-1 
systems,  to  plan,  monitor,  and  implement  aspects  of  the 
computer  resources  for  the  PM.  The  CRWG  is  an  integral 
part  of  the  PM's  acquisition  team.  It  is  established 
as  soon  as  computer  resources  are  determined  to  be  part 
of  the  system.  The  PM  is  responsible  for  chartering 
and  managing  the  CRWG. 

(2)  Membership  of  the  CRWG  includes  the  PM,  user 
representatives,  post-deployment  support  personnel 
(e.g.  LCSEC) ,  independent  testers  (developmental  and 
operational),  independent  evaluators  (developmental  and 
operational) ,  software  quality  assurance  (product 
assurance) ,  and  other  representatives  as  required. 

(3)  CRWG  duties: 

(a)  Manages  life  cycle  computer  resources  for 
the  system. 

(b)  Prepares  and  updates  the  system's 
computer  resources  management  plan  (CRMP)  throughout 
the  life  cycle.  This  document  describes  the  management 
strategy  for  software  development,  testing,  and  life 
cycle  support  and  is  required  lAW  AR  70-1. 

(c)  Provides  software  and  computer  resources 
expertise  to  the  TIWG. 

(d)  Provides  input  to  the  system  TEMP  through 

the  TIWG. 

(e)  Reviews  all  requests  for  proposal  (RFPs) 
to  ensure  that  software  T&E  factors  are  addressed.  A 
checklist  of  software  T&E  factors  to  be  addressed  is  in 
Chapter  8  of  this  part  of  DA  Pamphlet  73-1. 

(f)  Members  can  serve  on  the  Source  Selection 
Evaluation  Boards  (SSEB)  (except  the  independent 
operational  testers  and  operational  evaluators) .  SSEB 
members  review  the  bidders'  responses  and  assess  their 
capabilities  to  develop  quality  software. 

(g)  Monitors/participates  in  software 
develojpment  and  T&E  process. 

(h)  Provides  a  memorandum  of  agreement  (MOA) 
to  the  TIWG.  The  MOA  identifies  the  products, 
analyses,  and  T&E  support  to  be  provided  by  the  CRWG. 

(i)  Updates  the  TIWG  periodically  with  the 
status  of  software  T&E,  the  impacts  of  software 
deficiencies,  the  readiness  of  software  to  support 
further  T&E  events,  software  T&E  metrics  and  other 
related  factors. 

(4)  An  effective  CRWG  integrates  the  management 
of  computer  resources  for  the  PM  through  the  TIWG. 
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Chaptar  5 

Idantif ication  of  Naada  and  Davalopaant  of  Concapts  for 
syataa/softwara  TiE 

Saction  I 
Introduction 

5-1.  Ovarviaw 

System  needs  are  identified  as  operational  mission 
requirements.  These  requirements  are  the  result  of  an 
identified  operational  deficiency  or  the  potential  to 
increase  productivity  and  efficiency  within  an 
organization  or  process.  The  processes  of  identifying 
the  needs  and  evaluating,  selecting,  and  dociimenting  a 
system  concept,  which  Includes  software,  are  areas  in 
which  T&E  organizations,  especially  software  TiE 
organizations,  have  had  little  impact.  The  needs 
identified  at  this  time  are  not  tested  until  develop¬ 
ment  is  complete  and  the  end-item  is  the  system.  This 
chapter  discusses  two  crucial  phases  of  the  system  life 
cycle:  mission  need  determination  and  concept 
exploration/definition.  It  discusses  related  software 
test  and  evaluation  activities  which  occur  during  these 
phases. 

Section  II 

Mission  Meed  Determination 

5-2.  Early  requirements  documents 

Needs  are  identified  through  a  number  of  methods. 

Needs  may  be  the  natural  outgrowth  of  studies  and 
analyses  of  the  military  threat  posed,  a  response  to  a 
requirement  to  perform  a  new  mission,  a  response  to 
improvements  in  technology,  or  a  response  to  regulatory 
or  doctrinal  changes.  Once  the  need  is  identified  to 
the  extent  that  operational  users  and  the  decision¬ 
makers  create  a  proponency,  the  system  is  procured 
under  either  MSCR  (AR  70)  or  AIS  (AR  25)  policy.  TiE 
policy  for  any  system  is  lAW  AR  73-1. 

a.  Mission  needs  statement  (MNS) .  The  MNS  is 
developed  by  the  users'  representative,  functional 
proponent  or  combat  developer  (FP  or  CBTDEV) .  The  MNS 
is  the  official  starting  point  for  the  system  life 
cycle.  It  documents  the  need  identified  by  a  threat 
study,  an  information  requirements  study,  a  fix  to  a 
deficiency  identified  during  mission-area  analysis,  or 
a  regulatory  change. 

b.  T&E  involvement  during  mission  need 
determination.  T&E  involvement  at  this  time  deals  with 
broad  concepts  which  include  factors  such  as: 

(1)  Types  of  operational  testing  necessary. 

(2)  Scope  of  testing  rec[uired  based  upon  the 
acquisition  category. 

(3)  Identification  of  independent  evaluators. 
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c.  The  critical  operational  test  and  evaluation 
issues  and  measures  of  effectiveness  (MOE)  to  support 
the  eventual  MS  III  decision  are  developed  and  included 
in  the  TEMP  as  part  of  the  documentation  required  for 
MS  I.  By  necessity,  these  issues  and  MOE  are  based  on 
the  MNS  and  should  reflect  the  system's  ability  to 
enable  the  user  to  satisfy  the  mission  need.  The 
issues  and  MOE  are  developed  by  the  users' 
representative  in  coordination  with  the  operational 
evaluator.  They  are  approved  at  this  time  by  the  major 
command  overseeing  the  work  of  the  FP  or  CBTDEV. 

Section  III 

Concept  Exploration/Definition 

5-3.  Concept  exploration/definition  requirements 
documents  end  requirements  engineering 

As  alternatives  are  eliminated  during  concept 
exploration,  operational  concepts  and  system 
requirements  become  better  defined.  Several  activities 
occur  in  relation  to  the  management  aspects  of  the 
system  at  this  phase  of  the  life  cycle.  The 
requirements  documents  prepared  by  the  user 
representatives  further  define  the  overall  performance 
and  characteristics  at  the  system  level.  First,  the 
documents  driving  the  concept  definition  phase  will  be 
discussed,  followed  by  a  discussion  of  the  T&E 
relationships  and  procedures. 

a.  Operational  requirements  document  (ORD)  (applies 
to  AR  70-1  and  Class  I  AR  25-3  systems  only) . 

(1)  The  ORD  is  a  bridge  connecting  the  MNS  to 
the  acquisition  program  baseline  and  the  specifications 
for  the  concept  or  system.  At  each  milestone  (MS) 
decision  point  (starting  with  MS  I)  the  ORD  reflects 
the  current  state  of  evolutionary  requirements 
definition.  It  replaces  the  required  operational 
capabilities  (ROC) . 

(2)  The  user  or  user  representative  develops  an 
ORD  for  the  most  promising  system  concept.  At  MS  I, 
the  ORD  provides  objectives  and  minimxm  acceptable 
requirements  for  performance  capability  parameters 
necessary  to  characterize  the  proposed  system  concept. 
These  roll  into  the  concept  baseline  and  are  used  in 
the  TEMP  as  thresholds. 

(3)  The  ORD  is  used  to  develop  requirements  for 
the  draft  system  specification. 

(4)  During  this  period,  the  evolving  system 
definition  and  prototype  experience  causes  updates  and 
expansion  of  the  ORD.  The  user  or  user  representative 
updates  the  ORD  to  include  objectives  and  minimum 
acceptable  requirements  for  those  performnnce 
capability  and  performance  characteristic  parameters 
that  characterize  the  proposed  system  design  approach. 
The  target  date  for  achieving  operational  capability 
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should  also  be  identified  along  with  a  final  list  of 
critical  system  characteristics.  It  will  be  used  to 
develop  requirements  for  the  system  and  development 
specifications.  The  ability  of  the  system  to  satisfy 
perfcprmance  requirements  described  in  these 
specifications  will  be  verified  by  development  test  and 
evaluation  and  engineering  design  analyses. 

(5)  The  minimum  acceptable  operational 
performance  specified  in  the  ORD  is  used  to  refine  test 
criteria  for  operational  test  and  evaluation.  The 
issues  and  MOE  must  remain  mission  focused. 

Operational  test  and  evaluation  will  also  provide  data 
to  characterize  actual  system  performance  capabilities 
in  the  intended  operational  environment. 

(6)  After  MS  II,  the  ORD  should  only  be  modified 
as  a  result  of  a  change  in  the  MNS  or  cost-schedule- 
performance  trade-offs  during  development. 

b.  Users'  Functional  Description  (UFD) .  .. 

(1)  The  user  representative  (CBTDEV,  FP  or  PA) 
prepares  a  UFD  to  bridge  the  high-level  system 
initiation  documents  (MNS,  ORD)  to  the  detailed  system 
specifications.  The  UFD  provides  information  to 
communicate  the  operational  requirements  for  automated 
processing/computer  resource  capabilities  within  the 
system.  All  AIS  projects  have  a  UFD.  AR  70-series 
systems  with  automated  processing  capabilities  also 
have  a  UFD.  The  format  for  writing  a  UFD  is  contained 
in  DA  Pamphlet  XX-XX,  "Operational  Requirements  for 
Automated  Capabilities." 

(2)  The  UFD  defines  the  operational  requirements 
of  systems  in  sufficient  detail  to  guide  the  software 
development.  The  UFD  describes  implications  of  the 
operational  requirements  for  automated  capabilities  on 
the  system's  operational  modes  and  mission  profile, 
proposed  procedures,  interfaces  with  other  systems,  and 
degraded  operations.  It  amplifies  information  about 
the  system's  environment  as  it  impacts  the  system's 
automated  processes. 

(3)  System  requirements  are  frequently  divided 
into  bloclcs  which  represent  levels  of  performance  or 
functionality.  These  bloc)cs  may  form  incremental 
releases  which  lead  to  the  final  full-up  system. 

Bloclced  user  requirements  must  be  clearly  defined  and 
documented  to  ensure  adequate  T&E  for  releases  or 
formal  testing  for  production  decisions. 

(4)  The  UFD  can  be  in  draft  form  during  mission 
need  determination  and  concept  exploration/definition. 
It  is  expected  that  the  UFD  is  a  dynamic  document 
during  early  stages  where  there  are  many  un]cnowns. 

This  means  that  it  can  be  updated  as  frequently  as 
necessary  in  response  to  prototyped  versions  and  other 
demonstrations.  As  the  un]cnowns  are  understood,  the 
UFD  should  become  less  volatile. 
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(5)  The  UFD  should  be  baselined  and  approved  for 
AIS  25-series  systems  by  AIS  MS  I.  For  block 
developments,  each  block  should  be  approved  by  MS  I  for 
AIS.  The  UFD  for  AR  70-series  systems  should  be 
baselined  and  approved  by  MSCR  MS  II. 

c.  Requirements  engineering. 

(1)  The  user  representatives  use  concept 
exploration/definition  and  demonstration/validation 
phases  to  continually  refine  the  software  requirements. 
Varying  methods  may  be  applied  to  achieve  this. 

Examples  are  rapid  prototyping  and  user  demonstrations 
of  requirements.  These  methods  give  users  the  chance 
to  see  pieces  of  the  system  early  and  make  changes 
before  system  design  begins. 

(2)  User  demonstrations  are  useful  (and 
encouraged)  to  show  the  user  concepts  of  how  the 
requirements  might  be  implemented  through  software. 

The  better  the  user  eatresses  those  needs  in 
requirements,  the  easier  the  developer  can  design  the 
system.  This  also  assists  the  T4E  community  to  plan 
and  focus  T&E  efforts.  Early  user  demonstrations 
provide  the  opportunity  to  check  and  refine 
requirements. 

5-4.  Concept  exploration/definition  TSE  documents 
At  key  milestones  in  the  acquisition  process,  decision¬ 
makers  must  decide  whether  a  system  will  proceed  into 
the  next  phase  of  acquisition.  At  these  designated 
milestones,  developmental  and  operational  evaluators 
are  required  to  prepare  and  present  assessment  reports 
to  the  decision-makers,  if  there  is  no  IViV  agent 
appointed  for  the  acquisition  and  development,  the  PM 
will  report  that  fact  to  the  decision-makers.  A 
justification  for  not  using  IV&V  agents  during  software 
development  will  be  required.  For  cases  where  no  IViV 
agent  is  used,  the  developmental  and  operational 
evaluators  will  prepare  additional  reports  regarding 
software  development,  design,  and  traceability  to  the 
user's  requirements. 

a.  Critical  operational  issues  and  criteria  (COIC) . 
The  COICs  are  prepared  by  the  user  representatives  and 
coordinated  with  the  independent  operational  evaluator 
lAW  Part  Three  of  DA  Pamphlet  73-1.  Issues  take  the 
form  of  questions,  the  answers  to  which  permit  an 
evaluation  of  the  overall  operational  effectiveness  and 
suitability  of  the  system  (software).  Criteria  are 
thresholds  against  which  issues  are  evaluated.  These 
critical  issues  and  criteria,  once  approved,  are 
documented  in  Part  IV  of  the  system's  TEMP.  COICs  are 
approved  by  either  Deputy  Chief  of  Staff  Operations  and 
Plans  (DCSOPS)  or  Director  of  Information  Systems 
Command,  Control,  Communications  and  Computers  (DISC4) 
prior  to  MS  II  for  all  systems  managed  at  or  above  the 
ASARC  or  MAISRC  levels.  There  is  no  separate  set  of 
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COICs  for  software.  Issues  and  criteria  are  developed 
to  meet  the  following  general  objectives: 

(1)  Comprehensiveness  —  ensure  that  all 
concerns  which  impact  the  operational  use  of  a  system 
are  addressed  and  ensure  that  the  users  can 
satisfactcorily  accomplish  the  critical  mission 
functions  without  any  undue  burden. 

(2)  Realism  ~  ensure  that  system  operational 
characteristics  contained  in  the  requirements  documents 
are  translated  into  user  criteria  that  address 
operational  effectiveness  and  suitability. 

(3)  Progressiveness  —  ensure  that  the  criteria 
recognize,  where  appropriate,  that  the  development 
process  builds  on  technical  and  operational  learning. 

b.  A  TEMP  is  required  for  systems  under 
development.  An  approved  TEMP  is  required  by  Milestone 
I  and  includes  system  requirements  stated  in  the  ORD  or 
MNS.  TEMPS  for  systems  after  Milestone  III  are  as  .. 
required  I AW  AR  73-1. 

(1)  The  TEMP  identifies  the  T&E  organizations 
and  responsibilities,  strategy,  and  milestone  schedule 
for  the  system.  The  TEMP  also  identifies  issues  and 
criteria  for  evaluating  the  system  and  may  include  the 
use  of  metrics  as  described  in  Chapter  17,  Software  T&E 
Metrics,  of  this  part  of  DA  Pamphlet  73-1.  The  level 
of  completeness  for  the  technical  issues  and  criteria 
also  referred  to  as  technical  parameters,  will  nc- 
vary  with  the  maturity  of  the  system  as  it  moves 
through  its  life  cycle  process.  Satisfaction  of 
operational  characteristics  based  on  the  ORD  and  UFD 
can  be  demonstrated  in  technical  tests  or  in  early  or 
limited  user  tests.  These  demonstrations  of  user 
functional  capabilities  serve  the  program  sponsor  as  a 
measure  of  progress  toward  satisfying  the  mission  need 
in  the  lOT  required  for  a  successful  MS  III. 

Development  issues  and  criteria  can  be  satisfied 
through  a  variety  of  sources,  not  only  test  events. 
Other  sources  might  include  models,  simulation, 
surveys,  studies  or  other  means.  The  TEMP  format  is 
provided  in  Part  Two  of  DA  Pamphlet  73-1. 

(2)  If  the  accelerated  development/fielding 
acquisition  approach  is  to  be  used,  the  sets  of 
critical  mission  functions  comprising  each  software 
block  are  identified  as  part  of  the  critical  technical 
parameters.  The  critical  mission  functions/ blocks 
needed  to  provide  a  minimum  useful  operational 
capability  are  identified  as  the  representative  sample. 
The  representative  sample  is  the  first  aggregate  of 
blocks  suitable  for  fielding  to  users  after  it  is 
operationally  tested. 

c.  Independent  evaluation  plan  (lEP) /independent 
assessment  plan  (lAP)  and  test  design  plan  (TDP) .  The 
Government  developmental  lEP/IAP  and  the  TDP  are 
drafted  during  the  concept  exploration/definition 
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phase.  The  formats  for  these  plans  are  tailored  as 
required,  but  the  general  content  and  format  are  found 
in  Part  Four  of  DA  Pamphlet  73-1  regarding 
developmental  test  and  evaluation. 

d.  Test  and  evaluation  plan  (TEP) .  Operational 
tests  and -independent  operational  evaluations  use  the 
TEP  for  system  and  software  evaluation.  The  TEP 
describes  the  strate^  for  testing  and  evaluating  the 
system  COICs.  Additional  operational  issues  and 
criteria  may  be  included  as  well.  The  TEP  format  can 
be  found  in  Part  Five  of  DA  Pamphlet  73-1. 

e.  Acquisition  decisions  for  developmental  and  non- 
developmental  programs  are  heavily  influenced  by  the 
analysis  and  evaluation  of  data  associated  with  issues 
and  criteria.  Approved  issues  and  criteria  are  used  to 
determine  the  scope,  emphasis  and  intensity  of  the  T&E 
effort.  This  determination  is  the  basis  for  the 
resources  that  must  be  committed  to  obtain  the  data  to 
answer  the  issues  and  evaluate  the  degree  to  which 
criteria  are  met.  The  PM  and  T&E  community  are 
responsible  for  ensuring  that  the  required  resources 
are  programmed,  bud  ated,  and  available. 

f.  Technical  i£  es  and  criteria  (reference  Part 
Four  of  DA  Pamphlet  73-1)  are  developed  by  the 
developmental  evaluator  and  relate  to  critical 
technical  characteristics  or  parameters  of  system 
performance,  reliability,  maintainability,  software 
functions  and  features,  communications,  technological 
attributes,  as  examples.  Critical  technical  system 
issues  are  stated  in  the  developmental  lEP  or  lAP 
(reference  Part  Four  of  DA  Pamphlet  73-1) ,  and 
described  or  summarized  in  Part  III  of  the  TEMP. 

5-5.  TtE  resourcing  for  operational  testing 

a.  Operational  test  and  evaluation  funding.  Part  V 
of  the  TEMP  provides  a  summary  of  all  key  resources, 
both  Government  and  contractor  planned,  to  be  used 
during  the  course  of  the  acquisition  program.  The 
initial  TEMP  projects  those  key  resources  necessary  to 
accomplish  developmental  test  and  evaluation  (DT&E)  and 
operational  test  and  evaluation  (OT&E)  objectives, 
including  contractor  or  government  test  beds,  test 
drivers  (simulators  or  stimulators) ,  major  range  and 
unique  instrumentation  requirements,  threat  simulators, 
and  targets. 

b.  Outline  test  plan  (OTP) . 

(1)  The  OTP  provides  detailed  information  on 
resources  required  to  support  Part  IV  of  the  TEMP. 
Evaluation  plans  and  test  plans  integrate  test 
requirements  and  provide  the  audit  trail  for  validation 
of  resource  requirements.  The  OTP  is  the  basis  for 
programming  and  budgeting  for  tests  requiring  user 
troops.  The  OTP,  as  the  tasking  document,  is  usually 
revised  and  updated  based  on  test  requirements  when  the 
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TEP  and  test  plans  have  been  written.  The  documents 
are  compared  and  trade-offs  between  test  requirements 
and  available  resources  are  discussed.  The  result  is 
coordinated  docviments  which  meet  test  objectives  and 
consume  minimum  resources.  The  OTP,  as  updated, 
establishes  the  scope  and  resource  ceilings  within 
which  testing  will  be  conducted. 

(2)  Test  schedule  and  review  committee  (TSAPC) . 
The  TSARC,  using  information  from  OTPs,  provides  high- 
level,  centralized  Army  management  of  resources  (i.e., 
personnel,  equipment,  funding,  flying  hours,  ammunition 
and  instrumentation)  for  operational  test  and 
evaluation.  The  TEMP  is  reviewed  by  the  TSARC  to 
ensure  that  resource  requirement  updates  are 
accommodated  in  the  OTP. 

(3)  The  OTP  is  the  principal  resource  management 
document  used  by  the  TSARC  for  operational  testing. 

All  direct  costs  for  operational  .testing  are  delineated 
in  an  OTP.  It  lists  the  necessary  resources  and  the 
administrative  requirements  to  support  an  operational 
test  and  evaluation  (AR  15-38)  as  well  as  the 
associated  suspense  dates  and  test  milestones.  When 
included  in  the  approved  five-year  test  plan  (FYTP) ,  an 
OTP  becomes  a  formal  resource  tas}cing  document  for  test 
execution  and  resource  allocation  within  program  and 
budget  constraints.  These  constraints  are  in 
accordance  with  priorities  for  tests  scheduled  in  the 
current  and  budget  years.  Refer  to  Part  Four  of  DA 
Pamphlet  73-1  for  OTP  descriptions  and  procedures. 

5-6.  Evaluations  during  the  concept  exploration/ 
definition  phase 

During  concept  definition  the  software  T&E  community 
performs  evaluations  in  specialized  areas  related  to 
software.  The  software  T&E  community  will  be  heavily 
involved  in  reviewing  the  system-level  statement  of 
requirements  which  is  one  of  the  primary  outputs  of  the 
concept  definition  period.  There  are  three  types  of 
evaluations  discussed  here;  (l)  continuous  evaluation 
performed  by  any  organization,  (2)  independent 
developmental  evaluations/assessments,  and  (3) 
independent  operational  evaluations  performed  by 
specially  assigned  organizations.  All  of  these 
evaluations  provide  input  to  the  PM  and  decision- 
ma}cers. 

a.  Continuous  evaluation.  Evaluation  is  performed 
by  the  software  T&E  community  for  the  PM  and  decision- 
ma)cers.  Various  agencies  provide  this  information 
during  the  concept  definition  phase.  The  level  of 
technical  software  evaluation  varies  with  the  extent  of 
concept  definition  performed.  Continuous  evaluation 
talces  many  forms  during  the  system  concept  definition. 
It  is  important  for  the  software  T&E  community  to  be 
flexible  and  provide  the  level  of  evaluation  support 
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necessary  to  move  from  concept  phase  into  other  phases , 
as  well  as  provide  the  required  information  for 
decision-ma)cers  at  designated  milestones.  During 
concept  definition,  the  software  T&E  community 
participates  in  the  technical  evaluation  of  the  system 
and  Its  ability  to  support  operational  doctrine  and 
concept.  Early  operational  assessments  (EOAs)  are 
provided  to  the  program  sponsor  and  decision-makers 
during  this  phase.  The  scope  of  the  software  T&E 
community  involyement  depends  upon  the  approach  used  in 
concept  definition  and  the  acquisition  strategy. 

System  concepts  which  are  well-defined  and  concept 
risks  that  are  low  may  lead  to  less  activity  during 
this  phase.  System  concepts  which  are  to  be 
prototyped,  advanced  developed,  alternatives  to  be 
analyzed,  benchmarked,  and  fly-offs  or  run-offs  between 
differing  concepts  demand  more  activity  from  the 
software  T&E  community.  If  software  developments  occur 
during  concept  exploration/def inition  and  this 
development  is  intended  to  be  carried  over  into  the 
design  and  development  of  the  actual  system,  then  the 
software  T&E  community  will  be  heavily  involved  in  this 
phase.  Furthermore,  the  formal  documentation 
requirements,  design  reviews,  and  software  testing  are 
required  to  be  performed  for  the  software  to  be 
supportable  and  carried  over  into  the  next  phase.  If 
software  is  developed/prototyped  and  used  only  to 
demonstrate  the  concept  and  thrown  away,  the  software 
T&E  involvement  and  types  of  evaluation  are  less 
formalized. 


(1)  Documentation.  The  primary  software  and 
system  requirements  documents  are  usually  in  the  form 
?Lf  sections  of  the  functional  description 

(FD) ,  or  the  draft  system  level  specification, 
sometimes  called  the  system/ segment  specification  (SSS) 
or  A-specification.  These  require  thorough  review  and 
evaluation. 

Factors.  The  software  T&E  community 
reviews  the  system  requirements  documentation  as 
follows: 


...  Determines  if  the  requirements  as 

specified  can  be  tested,  and  if  the  requirements  have 
measurable  criteria. 

(^)  Determines  if  the  user  recruirements 
(ORD,  MNS,  UFD,  etc.)  trace  to  the  statement  of  system 
requirements.  This  is  normally  an  IV&V  function 
performed  for  the  PM. 

(c)  Determines  if  the  system  characteristics, 
system  performance,  peak  loading,  message  formats 
system  interfaces  (e.g.,  identifying  interfaces,  other 
systems  for  interoperability)  are  feasible. 

.  (<3)  Determines  if  the  requirements  stated  in 
the  various  levels  of  documents  are  consistent  with  one 
another . 
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(e)  An  overall  review  of  the  UFD  and/or 
system— level  specification  must  also  include  the  review 
of  trade-offs  and  risks  associated  with  the  allocation 
of  requirements,  and  review  the  trade-offs  for 
potential  operational,  technical,  or  T&E  impacts. 
Operational  and  developmental  evaluators  also  review 
the  capability  of  the  system  to  meet  the  technical 
requirements  and  operational  missions. 

(3)  If  the  acquisition  strategy  calls  for  an 
accelerated  software  development,  it  may  be  necessary 
to  conduct  more  stringent  evaluations  and  tracking  as 
described  in  Chapters  6  and  7  of  this  part  of  DA 
Pamphlet  73-1,  The  extent  of  these  evaluations, 
including  the  use  of  metrics,  are  dependent  upon  the 
degree  to  which  the  software  is  carried  over  in  future 
system  phases  and  the  delivery  requirements  of  the 
statement  of  work  (SOW) . 

(4)  SOW  reviews.  The  software  T&E  community 
provides  input  and  review  of  in-house  work  directives, 
RFPs,  SOWS,  and  other  work  tasks  relating  to  software 
development  and  T&E.  Depending  upon  the  type  of 
contracting  or  the  type  of  in-house  work  directive, 
software  T&E  personnel  (except  operational  testers  and 
operational  evaluators)  may  participate  in  SSEBs 
through  the  CRWG  or  through  the  TIWG.  A  checklist  for 
reviewing  these  documents  and  sample  contract  clauses 
are  provided  in  Appendix  B,  Statement  of  Work,  of  this 
part  of  DA  Pamphlet  73-1. 

(5)  Benefits  to  SOW  reviews.  The  long-term 
importance  of  this  activity  and  its  impact  on  software 
T&E  is  usually  not  addressed  until  there  is  a  problem 
in  the  contract  or  the  in-house  work  directive.  It  is 
imperative  to  ensure  that  contract  deliverables  are 
clearly  stated  and  timed  to  ensure  useful  information 
is  received  when  needed.  As  an  example,  all  open 
software  problem  reports  must  be  available  prior  to  the 
start  of  Government  testing.  The  T&E  people  review  the 
SOW  and  in-house  work  directives  to  make  sure  that  the 
following  issues  are  addressed:  software  design 
reviews;  collection  or  delivery  of  metrics;  delivery  of 
software  problem  reports  from  the  developers;  hooks, 
ports,  etc.  to  instrument  for  test  data  collection 
purposes;  and  delivery  of  T&E  or  other  software 
dociunent  at  ion . 

b.  Independent  developmental  evaluations/ 
assessments. 

(1)  Independent  developmental  evaluations/ 
assessments  are  accomplished  by  assigned  organizations 
lAW  AR  73-1  at  critical  acquisition  stages.  These 
evaluations  address: 

(a)  Basic  engineering/mathematical 
assessments  of  test  results, 

(b)  System  performance. 
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(c)  System  reliability,  availability,  and 
maintainability  (RAM) , 

(d)  Integrated  logistic  support  (ILS) , 

(e)  Manpower  and  personnel  integration 
(MANPRINT)  issues,  including  human  factors  engineering 
(HFE)  (e.g. ,  user-friendliness,  worlcloading)  ,  and 
system  safety. 

(f)  Implementation  of  metrics  as  defined  in 
Chapter  17,  Software  T4E  Metrics,  of  this  part  of  DA 
Pamphlet  73-1. 

(2)  These  evaluations/assessments  of  systems 
containing  software  can  ta)ce  the  form  of  paper  analyses 
or  a  complete  concept  evaluation  providing  formalized 
documentation  and  formalized  events  for  evaluation  such 
as  design  reviews  and  actual  test  events. 

(3)  Independent  developmental  evaluations/ 
assessments  performed  during  concept  definition  are 
structured  to  answer  or  partially  answer  specific  high- 
risk  issues  and  criteria.  These  are  issues  which  have 
been  identified  as  technically  risky  due  to  use  of 
emerging  technology,  requirements  for  real-time 
processing  that  push  state-of-the-art  capabilities, 
requirements  for  innovative  interactive  interfaces, 
application  of  artificial  intelligence  (AI)  technic 

to  information  processing,  or  the  application  of  an 
existing  technology  to  a  new  area. 

c.  Independent  operational  evaluations  and 
operational  testing.  Independent  operational 
evaluations  are  performed  by  specially  assigned 
organizations  lAW  AR  73-1  during  the  concept  definition 
process  and  throughout  the  life  cycle.  During  concept 
definition  an  EOA  is  prepared  lAW  DA  Pamphlet  73-1  Part 
Five,  which  relies  heavily  upon  technical  data.  It 
also  focuses  on  mission  performance,  operational 
employment,  and  issues  of  suitability  of  the  concepts. 
The  EOA  is  planned  in  conjunction  with  an  abbreviated 
TEP  and  answers,  in  full  or  in  part,  some  set  of  issues 
and  criteria.  The  EOA  will  form  the  basis  of  the 
report  to  the  decision  makers  for  the  next  milestone 
decision.  It  may  be  in  the  form  of  a  strictly  paper 
analysis  or  it  may  be  the  result  of  early  concept 
testing  on  advanced  development  models,  prototypes  or 
several  competing  concepts. 

(1)  Early  operational  evaluation.  Results  from 
early  user  testing  or  other  operational  tests  are 
evaluated  and  reported  in  the  EOA.  The  EOA 
specifically  addresses  software  from  two  perspectives: 

(a)  Is  the  software  being  designed  with  the 
user  in  mind?  Can  the  user  interface  effectively  with 
the  software  and  consistently  control  it  so  that  the 
required  system  functions  are  provided? 

(b)  Is  the  software  "on-track"  with  respect 
to  planned  state  of  development?  In  particular,  are 
there  any  indications  that  the  software  will  not  be 
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ready  to  support  the  objectives  of  further  operational 
testing? 

(2)  In  addition,  problems  experienced  with  the 
software  and  observations  about  the  utility  of 
developed  test  tools  may  be  reported.  The  version  of 
the  software  which  was  tested  is  described  along  with 
the  significance  of  any  deviations  from  the  software  to 
be  fielded. 

(3)  Operational  testing.  Operational  testing  at 
this  phase  may  be  suggested  by  the  user 
representatives.  Early  in  concept  formulation, 
operational  tests  may  take  the  form  of  Early  User  Test 
and  Experimentation  (EUTE) ,  Limited  User  Test  (LUT) ,  or 
Force  Development  Test  and  Experimentation  (FDTE)  as 
described  in  AR  73->l.  Results  of  these  tests  are  used 
to  evaluate  the  proposed  concept(s),  refine  user 
requirements,  provide  input  for  concept  selection, 
provide  input  for  milestone  decisions  and  assist  in 
refining  tactics,  doctrine  or  standard  operating 
procedures  (SOP).  Detailed  explanation  of  these  test 
types  are  in  DA  Pamphlet  73-1,  Part  Five. 

(4)  Evaluation  reporting.  Evaluations  provide 
formal  scheduled  assessments  of  the  system's  and 
software's  operational  effectiveness  and  suitability  to 
decision-makers  at  milestone  decision  reviews  and 
provide  periodic,  ad-hoc  status  reports  whenever 
changes  occur  in  the  status  of  the  evaluation. 
Independent  evaluations  are  produced  in  the  following 
three  forms.  They  are  described  in  detail  in  Part  Five 
of  DA  Pamphlet  73-1: 

(a)  Independent  evaluation  briefings  (lEB)  to 
milestone  decision  reviews. 

(b)  Formal  test  and  evaluation  reports 

(TERs) . 

(c)  Operational  assessments  (OA,  EOA,  AOA) . 

(5)  Operational  assessments.  The  OA  is  an 
abbreviated,  less  formal  report  of  independent 
operational  findings.  The  OA  must  be  compatible  with 
the  lEB.  An  OA  may  be  an  early  OA  (EOA),  i.e.,  prior 
to  MS  II ,  or  an  abbreviated  OA  (AOA) ,  which  is  produced 
instead  of  an  TER  when  scheduling,  testing  limitations, 
decision-maker  requirements,  or  other  factors  preclude 
a  comprehensive  operational  evaluation.  In  this  case 
the  operational  testing  accomplished  is  typically  FDTE, 
EUTE,  concept  evaluation,  or  similar  developmental  test 
event. 

(6)  Independent  evaluation  briefing.  Formal 
lEBs  are  presented  to  the  decision  bodies  by  the 
operational  evaluator  at  each  major  decision  milestone. 
The  briefing  summarizes  the  evaluation  (OA  or  TER) 
submitted  prior  to  the  Defense  Acquisition  Board  (DAB) , 
Army  Systems  Acquisition  Review  Council  (ASARC) , 
In-Process  Review  (IPR)  Panel,  or  Major  Automated 
Information  System  Review  Council  (MAISRC) ,  and 
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contributes  to  management  decisions  by  the  review  body. 
The  format  for  an  lEB  is  provided  in  Part  Five  of  DA 
Pamphlet  73-1.  Most  often,  the  independent  operational 
evaluator  presents  the  key  technical  findings  of  the 
developmental  evaluation  along  with  the  operational 
evaluation. 


5-7.  Metrics  during  mission  need  datarmination  and 
concept  axploration/dafinition 

The  metrics  with  a  check  mark  apply  at  this  time: 


/  Cost 
/  Schedule 
CRU 
SEE 


Reqts  Traceability 
Reqts  Stability 
Design  Stability 
Complexity 


Breadth  of  Testing 
Depth  of  Testing 
Fault  Profiles 
Reliability 


a.  Collection  of  data  for  the  cost  and  schedule 
metrics  is  started.  The  metrics  are  examined  for 
adherence  to  allocated  budgets  and  planned  events. 

b.  Preliminary  traces  of  user  and  system 
requirements  begin.  These  traces  include  MNS/ORD  to 
UFD  and  UFD  to  draft  FD  or  SSS.  For  systems  undergoing 
operational  testing  in  these  eerly  phases,  requirements 
will  likely  be  more  comprehen  ive  and  the  traces  more 
complete.  In  this  case,  acceptable  degrees  of 
requirements  traceability  and  stability  may  be 
appropriate  as  exit  objectives. 


5-8.  Decision  criteria 

a.  The  outputs  from  the  two  phases  described  in 
this  chapter  must  provide  the  basis  for  software 
design,  development,  software  T&E,  and  ultimately 
system  TiE.  Refer  to  Table  5-1  for  decision  criteria. 

b.  Decisions  to  move  into  the  system  design  and 
software  design  phases  .are  the  results  of  milestone 
reviews.  A  positive  milestone  review  results  in  the 
commitment  by  the  Army  to  move  forward  with  the 
selected  concept  design  and  to  develop  the  system  and 
software. 
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Tabla  5-1.  Mission  Nssd  Dstsraination  and  Concept 
Exploration/Definition  Products 

Priaary 


Raaponaibility 

Products 

Decision  Criteria 

User  Representative 
(e.g.,  CBTDEV,  FP, 

PA) 

Critical  Operational 
Issues  t  Criteria 
(COIC) 

Approved 

Mission  Needs 

Statement  (MNS) 

Approved 

• 

Operational  Require¬ 
ment  s  Document  ( ORD ) 

Approved 

Users '  Functional 
Description  (UFD) 

Drafted 

PM  and  TIWG 

Test  6  Evaluation 
Master  Plan  (TEMP) 

Approved 

PM  and  CRWG  * 

(AR  70-1  systems 
only) 

Computer  Resource 
Management  Plan 
(CRMP)  • 

Drafted 

PM  and  Materiel 
Developer 

System/Segment  Spec. 
(SSS)  or  Functional 
Description  (FD) 

Drafted 

Independent  Develop¬ 
mental  Evaluator 
or  Assessor 

Critical  Technical 
Character. /Parameters 

Critical  Technical 
Issues  and  Criteria 

Included  in  approved 
TEMP 

Drafted  in  lEP  or  lAP 

lER/IAR  * 

Completed  lER/IAR 
for  milestone  reviews 

Independent 

Operational 

Evaluator 

Additional  Operational 
Issues  &  Criteria 
(AOIC) 

Drafted  in  TEP 

BOA,  OA,  TER,  lEB  * 

Completed  for 
milestone  reviews 

Independent 

Operational 

Evaluator  and 
Operational  Teeter 

Test  and  Evaluation 
Plan  (TEP) 

Abbreviated  TEP  * 

Draft  TEP 

Completed 

Operational  Tester 

FDTE,  EUTE,  LUT  * 

Completed  Test  Report 

System  or  Software 

Developer, 

Independent 

Evaluator 

Requirements  Trace(s) 

Metrics  Report (s) 

Drafted 

Acceptable  degrees  of 
requirements 
traceability  and 
stability 

•  Program  dependent 
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chaptar  6 

Softvara  TiE  Activitlas  in  Halation  to  Softvara 
Raquiraaants  Dafinition  and  Daaign 

Saction  I 

Introduction  to  Dafinition  and  Daaign  Aetivitias 
6-1.  Introduction 

a.  Test  and  evaluation  activities  discussed  in  this 
chapter  are  designed  to  be  flexible  to  acconinodate 
various  acquisition  strategies.  The  design  activities 
performed  during  preliminary  and  detailed  design 
activities  may  be  cyclic.  Sometimes  designs  of  parts 
of  the  software  are  simpler  to  detail  and  engineer  than 
others.  This  process  allows  for  repetitions  of 
preliminary  and  detailed  design  activities  for  as  many 
cycles  as  necessary.  These  procedures  provide  the 
flexibility  to  move  forward  for  those  design  pieces 
which  are  ready  and  meet  the  criteria  for  the  next 
phase. 

b.  The  user  representatives  need  to  participate  in 
these  activities.  User  participation  will  ensure  that 
the  software  evolves  to  support  the  system  mission. 

User  demonstrations  of  capability,  walk-throughs  of  the 
user-machine  interfaces,  and  participation  in  the 
cyclic  design  reviews  are  examples  of  necessary  user 
participation. 

c.  This  chapter  describes  software  T&E  support 
activities  which  are  performed  for  continuous 
evaluation  purposes  for  the  PM.  These  activities  are 
designed  to  reduce  risks  and  provide  the  PM  with 
information  to  make  informed  decisions.  The  PM 
controls  these  phases  in  the  process  and  uses  software 
T&E  inputs  to  assist  in  disciplining  and  managing  the 
process. 

d.  Reviews  are  conducted  to  assess  the  readiness  of 
the  concepts  and  designs  to  move  from  one  level  of 
detail  to  the  next.  The  scope  of  the  reviews  depends 
upon  acquisition  type,  system  complexity,  and  the  PM  or 
MATDEV's  needs.  Reviews  are  conducted  by  the  PM  or 
MATDEV  and  matrix  support. 

e.  If  there  is  no  IV&V  agent  assigned  to  the 
program,  the  developmental  and  operational  evaluators 
may 

be  required  to  prepare  and  present  assessment  reports 
at  the  CDR  and  walk-throughs. 

f.  Decision  criteria  discussed  in  this  chapter  are 
used  as  standards  applied  to  decision-making.  Moving 
from  one  phase  to  the  next  is  a  decision  process 
executed  by  the  PM  or  MATDEV,  who  uses  this  information 
to  assess  system  status  and  determine  the  risk 
associated  with  progressing  to  the  next  phase. 
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Saetion  II 

Systams  Raquiramants  Analysis 

(-2.  Systam  raquiramants  analysis  phasa 

a.  RFPs,  SOWS  and  in-house  work  directives,  top- 
level  documentation  of  the  system  requirements  concept 
in  the  UFD,  FD  or  SSS  aid  in  selecting  the  software 
developer.  Developers  may  be  in-house  or  may  be 
contractors. 

b.  The  set  of  activities  in  which  the  software  T4E 
community  participates  is  a  series  of  working-level 
reviews  of  the  higher  system-level  requirements.  These 
requirements  are  decomposed  and  allocated  by  the 
developer  to  more  detailed  functions  and  features  to 
support  the  overall  system  concept. 

6-3.  T6E  activities  -  systams  raquiramants  analysis 

a.  System  requirements  analysis  is  the 
responsibility  of  the  PM  and  developer.  It  is  used  to 
determine  whether  preliminary  requirements  allocated  to 
software  (SW)  and  hardware  (HW)  make  sense. 

Incremental  reviews  with  the  developer  using  informal 
(action  officer  level)  methods  (e.g.,  requirements 
walk-throughs)  ensure  that  requirements  in  the  MNS, 

ORD,  UFD,  FD  or  SSS  are  understood  and  addressed. 
Software  developers,  and  IV&V  agents  should 
begin/continue  implementing  the  requirements 
traceability  metric  as  defined  in  Chapter  17,  Software 
TiE  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

b.  Continuous  evaluation  activities  are  performed 
to  assess  the  requirements  and  provide  an  information 
base  for  evaluation  and  future  test  events.  The 
program  sponsor  reviews  the  system  requirements,  as 
follows,  with  the  software  T&E  community  providing 
support  to  items  (2),  (3),  (6),  (8)  and  (9): 

(1)  Determines  if  the  requirements  provide  a 
base  for  developing  system-level  design. 

(2)  Determines  if  the  requirements  in  the  FD, 

UFD  or  SSS  have  their  base  in  the  mission  needs  MNS  or 
ORD;  also  determines  whether  the  requirements  are 
satisfied  in  the  FD  or  SSS  by  performing  requirements 
traces. 

(3)  Determines  consistency,  in  terms  of  formats, 
protocols,  etc.,  through  review  of  initial  interface 
and/or  interoperability  requirements. 

(4)  Assesses  the  adequacy  of  the  stated 
requirements  to  support  software  development. 

(5)  Ensures  that  the  stated  design  will  achieve 
performance  goals  through  modeling  and  simulation 
activities. 

(6)  Determines  if  the  requirements  are  complete, 
unambiguous,  comply  with  stated  standards,  and  are 
testable  and  feasible. 


6-2 


30  8«pt«Bb«r  1992 


DA  Paa  73-1,  Part  Savan  (Draft) 


(7)  Determines  compliance  with  DOD  higher  order 
language  (HOL)  policy.  Ada  is  the  designated  HOL. 

(8)  Reviews  security  requirements. 

(9)  Reviews  post-deployment  support  alternatives 
for  operational  and  support  concepts. 


6-4.  Metrics  during  systan  raquiramaats  analysis 

The  metrics  with  a  check  mark  apply  at  this  time: 


/  Cost 
/  Schedule 
CRU 
SEE 


/  Reqts  Traceability 
✓  Reqta  Stability 
Design  Stability 
Complexity 


Breadth  of  Testing 
Depth  of  Testing 
Fault  Profiles 
Reliability 


a.  Cost  and  schedule  metrics  are  examined  for 
adherence  to  allocated  budgets  and  planned  events. 

b.  As  requirements  traces  are  updated  to  reflect 
evolving  requirements,  traceability  and  stability  of 
high  level  system  functions  can  be  assessed. 


6-5.  System  requirements  review  and  decision  criteria 
The  culmination  of  these  activities  is  a  formalized 
system  requirements  review  (SRR) .  The  review  logically 
is  a  wrap-up  of  the  informal  walk-throughs  conducted  at 
the  action  officer  level.  These  reviews  address  issues 
stated  in  MIL-STD-1521B  and  other  program  relevant 
issues. 

a.  Decision  criteria  (see  Table  6-1)  for  the  SRR 
need  to  be  considered  carefully  by  the  PM  in 
preparation  for  entering  the  system  design  phase.  The 
SRR  constitutes  a  review  of  mission  and  requirements 
analyses,  developer's  interpretation  of  requirements 
(e.g.,  FD,  SSS,  ORD) ,  functional  flows,  requirements 
allocation,  trade-offs,  system  interfaces,  initial  test 
planning,  technical  performance  measures  planning, 
post-deployment  support  concepts,  and  fielding  plans. 

b.  The  SRR  is  performed  by  the  PM,  supported  by  the 
developer,  and  other  support  groups.  Participants  in 
the  reviews  (e.g.,  IViV  agent,  user  representatives, 

SQA,  etc.)  need  to  provide  feedback  to  the  PM  in  the 
form  of  results  from  informal  walk-throughs,  system 
requirements  issues  and  their  impacts.  This  feedback 
is  part  of  an  overall  systems  engineering  principle  to 
ensure  that  deficiencies  are  addressed  early,  risks 
noted  and  decisions  made. 

c.  Decisions  to  move  to  the  next  phase  need  to 
consider  that  the  documents,  evaluations,  degree  of 
requirements  traceability,  and  reviews  show  that  these 
items  are  ready  to  support  the  next  phase  in  design/ 
development. 
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Tabla  6-1.  Systam  Raquiramants  Analysis 


Priaary 


Rasponsibillty 

Products 

Decision  Criteria 

User  Representative 
and  MATDEV 

Users*  Functional 
Description  (UFD) 

Final 

PM  and  MATDEV 

3y*tem/Segment  Spec. 
(SSS)  or  Functional 
Description  (FD) 

Updated  draft 

System/Subsystera 
Specification  (SS) 

Drafted 

PH  and  Developer 

[Configuration 
Management  Plan 

forafted 

SRR  Conducted 

System  or  Software 
Developer, 

SQA,  IVfiV  and 
Independent 

Evaluator 

Requirements  Trace(s) 

Metrics  Report (s) 

Updated 

Acceptable  degrees  of 
system  requirements 
etaf  ;  ity  and 
tra  ability 

Saetion  III 
Systam  Dasign 

6-6.  systam  dasign  activitias 

a.  System  design  is  the  responsibility  of  the 
system  developer.  The  system  design  provides  the 
engineering  solution  to  the  system  recpiirements. 

valuation  activities  performed 

(1)  Rgvxgv  sys^Gm  dGsxQn  docuiDGnfcs  vts 

SS.  draft  SRS,  US,  IRS,  9S)  for  the  “fifSSJSI:  ‘  ®®' 

;S!  design  provides  growth  capability. 

r«juire»eits.  '=°  ‘h* 

r«quire».«i  interoparability 

(d)  Review  provides  user-machine  interfaces 
trace  to  user-defined  requirements. 

(2)  Review  developer's  software  quality 
assurance  program  to  determine  developer's  internal 
evaluation  mechanisms. 

Review  developer's  proposed  software 
JSSutreroiLK'  cooplianoe  with 

(4)  Implementation  of  applicable  metrics. 
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(5)  Review  use  of  NDI  software  in  the  design 
considering  percent  of  modification,  availability  of 
documentation,  and  testability  issues. 

(6)  Review  use  of  software  development  tools  for 
capability  to  accomplish  a  quality  design. 

c.  Results  of  continuous  evaluation  assessments  are 
provided  to  the  PM  for  tracking  system  design  status. 

d.  The  formal  end  of  this  activity  is  the  system 
design  review  (SDR) .  This  review  should  be  a  roll-up 
of  the  design  addressed  informally  during  the  walk¬ 
throughs.  The  outcome  of  the  system  design  is  a 
functional  baseline  and  the  overall  system  concept  is 
completed. 

6-7 .  Metrics  during  system  design 

The  metrics  with  a  check  mark  apply  at  this  time: 

/  Reqts  Traceability  Breadth  of  Testing 

/  Reqts  Stability  Depth  of  Testing 

Design  Stability  Fault  Profiles 

Complexity  Reliability 

a.  Initial  processor,  memory  and  I/O  utilization 
budgets  should  be  available  at  this  time. 

b.  The  SEE  metric  applies  as  one  of  the  factors  in 
evaluating  software  contractors  if  one  or  more  vendors 
are  to  be  selected  at  this  time. 

6-8.  System  design  review  and  decision  criteria 

Decision  criteria  for  this  phase  are  shown  in  Table  6- 
2 . 

a.  The  SDR  is  performed  by  the  PM,  supported  by  the 
developer,  and  other  support  groups.  Participants  in 
the  reviews  (e.g.,  IV&V  agent,  user  representatives, 

SQA,  etc.)  need  to  provide  feedback  to  the  PM  in  the 
form  of  results  of  informal  walk-throughs,  system 
requirements  issues  and  their  impacts.  This  feedback 
is  part  of  an  overall  systems  engineering  principle  to 
ensure  that  deficiencies  are  addressed  early,  risks 
noted  and  decisions  made.  To  be  considered  successful, 
the  SDR  will  include  reviews  of  completed  SSS  or 
completed  FD  sections,  and  draft  software  requirements 
specifications  (e.g.,  SSR,  iRS,  SS,  database 
specifications  (DS) ,  unit  specifications  (US) ) .  The 
SDR  will  include  reviews  of  traceability  from  the 
system  level  specifications  back  to  the  UFD,  MNS  or 
ORD,  and  traceability  from  the  system  specifications 
into  the  draft  software  specifications.  At  this  point 
the  functional  baseline  for  the  system  is  established 
and  placed  under  configuration  control. 

b.  The  results  of  the  requirements  traceability, 
requirements  stability  should  be  presented  at  the  SDR. 
The  requirements  should  show  an  acceptable  degree  of 
completeness,  traceability  (SSS  to  SRS  and  IRS,  FD  to 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 
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Tabla  6-2.  Syataa  Dasign 


Primary 


Responsibility 

Products 

Decision  Critsria 

Developer  and  PM 

SSS 

Final  SSS 

SRS,  SS,  US 

Draft  SRS,  SS,  US 

IRS,  DS 

Draft  IRS,  DS 

Software  Development 
Plan  (SDP) 

Approved  SDP 

Software  Quality 
Program  Plan  <SQPP) 

Approved  SQPP 

Configuration 
Management  Plan 

Approved  CM  Plan 

SDR  conducted 

Functional  baseline 
established 

S/W  Developer 
and  Gov ' t .  SQA 
or  IV&V 

Rei^uirements  Trace  (a) 

Metrics  Report (s) 

Updated 

Acceptable  degrees  of 
requirements  traceability 
and  stability 

SS) ,  and  stability  (SSS  and  SS/FD) .  Preliminary  CPU 
allocations  to  computer  software  configuration  items 
(CSCIs)  should  be  compliant  with  the  contract  and 
higher-level  specifications. 

c.  Before  moving  to  the  next  phase,  the  documents, 
evaluations,  metrics  indicators,  and  review  should  show 
that  these  items  are  ready  to  support  the  next  phase  in 
design/development  or  risks  are  identified. 

Section  IV 

Software  Specification/Reguirementa  Analysis 

6-9.  software  specification/requirements  analysis 
activities 

a.  The  software  requirements  evolve  from  the 
approved  system  design.  The  software  requirements/ 
specifications  are  documented  in  software  requirements 
specifications  (SRS) ,  IRS,  or  DS,  US,  and  SS. 

b.  Continuous  evaluation  activities  performed 
within  the  T&E  community  need  to  include  user 
representatives.  Activities  performed  are: 

(1)  In-depth  software  requirements  reviews  of 
SRS,  IRS  or  DS,  US,  SS  for  testability,  performance, 
timing  and  interfaces. 
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(2)  Tracing  from  software  requirements  back  up 
to  system  requirements.  The  combat  developer  or 
functional  proponent  participates  in  the  requirements 
trace  to  ensure  that  the  users'  requirements  have  been 
properly  interpreted  by  the  software  specification 
writers.  - 

(3)  Review  of  user-machine  interfaces  for 
simplicity,  logical  sequencing,  and  tracing  back  to 
user-defined  requirements. 

(4)  Implementation  of  applicable  metrics. 

c.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  placing  the  set  of 
qualification  requirements  for  each  software  item  under 
control.  The  qualification  requirements  are  normally 
documented  in  the  set  of  development  specifications, 
subject  to  evaluation  for  adequacy  of  test  coverage  and 
testability,  such  as  the  SRS,  IRS,  or  SS.  The 
configuration  management  function  may  also  conduct  SSRs 
of  these  documents  prior  to  authentication.  ’  After 
authentication  of  the  set  of  software-related 
development  specifications  (which  are  entered  into  the 
allocated  baseline)  all  changes  to  these  documents  are 
processed  as  engineering  change  proposals  (ECPs) , 
reviewed  by  the  Army  Program/Project  Office 
Configuration  Control  Board  (CCB) ,  and  approved  by  the 
CCB  chairman. 


6-10.  Metrics  during  software  specif ieation/require- 
nents  analysis 

The  metrics  with  a  check  mark  apply  at  this  time: 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


/  Reqts  Traceability 
/  Regta  Stability 
Design  Stability 
Complexity 


Breadth  of  Testing 
Depth  of  Testing 
Fault  Profiles 
Reliability 


a.  The  SEE  metric  applies  as  one  of  the  factors  in 
evaluating  the  capabilities  of  various  software 
contractors . 

b.  Additional  information  on  metrics  as  applied  in 
this  phase  can  be  found  in  Chapter  17,  Software  T&E 
Metrics,  of  this  part  of  DA  Pamphlet  73-1. 


6-11.  Software  specif ication/rsquirsasnts  analysis 
decision  criteria 

Decision  criteria  for  this  phase  are  shown  in  Table  6- 
3. 

a.  The  formal  culmination  of  these  activities  is  a 
series  of  software  specification  reviews  (SSRs).  As 
previously  stated,  the  preferred  methodology  is  to  use 
a  series  of  incremental  and  informal  walk-throughs  and 
reviews  to  work  through  the  information  of  the  software 
requirements.  This  is  particularly  crucial  if 
requirements  are  still  evolving  and  incremental 
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®”3.  Boftvara  Bpacif ieation/Raquiramants  Analysis 


Priaary 


Responsibility 

Products 

Decision  Criteria 

Developer  &  PM 

SRS,  US 

IRS,  DS 

Test  Plans 

Pinal  SRS,  US 

Final  IRS,  OS 

Draft  Test  Plans 
(software  test  plans) 

SSR(a)  conducted 

Allocated  baseline 
established 

Oper.  Teeter  & 
Independent 

Oper.  Evaluator 

TEP 

Draft  TEP 

Devel.  Tester  & 
Independent 

Oevel.  Evaluator 

'lEP/IAP 

Updated 

PM/SW  Developer, 
Gov't  SQA 

Requirements  Trace(s) 

Updated 

or  IV&V,  and 
ueers  *  represen¬ 
tative 

Metrics  Report (s) 

Acceptable  degrees  of: 
requirements 
traceability  and 
stability; 

CRU  allocations 

development  occurs. 

b.  The  SSRs  are  performed  by  the  PM,  supported  by 
the  developer,  and  other  support  groups.  Participants 
in  the  reviews  (e.g.,  IV&V  agent,  user  representatives, 
SQA,  etc.)  need  to  provide  feedback  to  the  PM  in  the 
form  of  results  of  informal  walk-throughs,  system 
requirements  issues  and  their  impacts.  This  feedback 
is  part  of  an  overall  systems  engineering  discipline  to 
ensure  that  deficiencies  are  addressed  early,  risks 
noted  and  decisions  made. 

c.  Decisions  to  move  to  the  next  phase  need  to 
consider  that  the  documents,  evaluations,  metrics 
indicators,  and  review  show  that  these  items  are  ready 
to  support  the  next  phase  in  design/development. 

d.  The  results  of  the  requirements  traceability, 
requirements  stability,  and  CRU  metrics  should  be 
presented  at  the  SSR.  The  requirements  should  show  an 
acceptable  degree  of  completeness,  traceability,  and 
stability.  The  design  typically  would  show  some 
volatility  and  fluctuation  at  this  time.  CRU 
allocations  to  CSCIs /programs  should  be  compliant  with 
the  contract  and  higher  level  specifications.  If 
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requirements  show  volatility  or  if  traceability 
problems  exist,  it  may  be  premature  to  move  to 
preliminary  design. 

Section  V 

Preliminary  Design  Phase 
6-12.  Introduction 

The  preliminary  design  phase  can  be  described  simply  as 
the  developer's  efforts  to  further  allocate  the 
allocated  baseline  software  requirements  to  a 
preliminary  software  design.  Allocation  during  this 
phase  is  the  effort  to  better  define  (to  a  lower  level 
of  detail)  the  functionality  of  the  software,  so  that 
eventually  it  can  be  reduced  to  logic  flows  and  source 
code  in  later  design/development  phases.  What  results 
is  a  well-defined  high-level  design  architecture,  and 
when  determined  to  be  mature,  a  preliminary  design 
review  (PDR)  is  held  to  evaluate  and  approve  the  early 
design  prior  to  proceeding  to  the  next  phase.  It  is 
important  to  note  that  full  maturity  is  not  necessary 
to  get  approval  at  PDR,  since  the  developer  is  allowed 
to  have  the  flexibility  to  refine  and  optimize  the 
design.  However,  the  allocated  baseline  must  be  mature 
enough  to  allow  for  an  intelligent  decision  to  proceed 
forward  —  this  is  very  important  both  in  minimizing 
program  cost  and  schedule  ris)cs,  and  in  improving  the 
overall  quality  of  the  design.  For  complex  systems, 
the  PDR  may  require  a  series  of  iterative  reviews  as 
the  design  progresses  and  matures. 

6-13.  Preliminary  design  activities 

a.  There  are  three  primary  objectives  which  must  be 
achieved  during  preliminary  design.  Objective  one 
requires  the  demonstration  of  the  compatibility  between 
the  allocated  baselined  software  requirements  and  the 
proposed  preliminary  software  design.  The  primary 
method  to  verify  this  objective  is  through  an 
assessment  of  the  traceability  of  the  software 
requirements  down  to  the  proposed  preliminary  software 
design.  Both  the  US  and  DS  or  the  software  design 
document  (SDD  Part  I)  are  utilized  to  docximent 
preliminary  software  designs  for  Army  systems.  A 
traceability  assessment  should  demonstrate  that  all 
software  requirements  have  been  allocated  to  the 
proposed  preliminary  software  design.  This 
traceability  assessment  should  also  ensure  that  no 
extraneous  design  has  been  introduced  which  cannot  be 
traced  bacJc  to  the  baselined  software  requirements. 

This  traceability  assessment  should  include 
traceability  of  interface  requirements  defined  in  the 
IRS  or  FD  and  SS. 

b.  Objective  two  requires  the  demonstration  of  the 
technical  adequacy  of  the  proposed  preliminary  software 
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design.  The  primary  method  to  verify  this  objective  is 
through  an  assessment  of  the  capability  of  each 
preliminary  design  function /component  to  perform  the 
applicable  software  requirements.  The  DS,  US  or  SDD 
Part  I  represents  the  preliminary  software  design.  An 
assessment  of  each  function/component  should  examine 
what  the  software  is  supposed  to  do  as  stated  by  the 
software  requirements  and  how  the  software  will  perform 
those  requirements  as  described  in  the  contents  of  the 
US,  ps  or  SDD  Part  I.  An  assessment  of  this  type 
provides  insight  into  the  technical  adequacy  of  the 
proposed  preliminary  software  design. 

c.  Objective  three  calls  for  the  presentation  of 
the  proposed  software  test  plans  and,  if  applicable, 
the  proposed  user's  manual.  Test  plans  are  defined  and 
these  documents  identify  the  specific  methods  which 
will  be  utilized  to  test  all  levels  of  the  software 
system.  An  assessment  should  be  made  to  determine  if 
the  proposed  test  plan  can  provide  testing  which  will 
produce  the  information  needed  to  evaluate  software 
performance  and  validate  software  requirements.  The 
proposed  test  plan  should  be  an  explicit  document  'hich 
adequately  addresses  all  aspects  of  software  test'  g, 
including  stress  testing,  interface  testing,  regression 
testing,  and  built-in  test  validation. 

d.  Continuous  evaluation  activities  during 
preliminary  design  will  provide  reports  and  information 
to  the  PM.  The  objectives  for  preliminary  design  will 
be  assessed  and  reported  by  SQA,  IVSV  and  other 
participants  in  CE.  The  implementation  and  evaluation 
of  applicable  metrics  are  part  of  CE. 

e.  The  user  representatives  need  to  participate  in 
the  reviews.  Preliminary  design  emulations,  rapid 
prototyping  or  similar  methods  can  be  used  to 
demonstrate  the  early  designs  to  the  user. 

f.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  identifying  and 
controlling  the  test  planning  documents  and  test 
requirements.  The  configuration  management  function 
also  establishes  a  developmental  configuration  that 
stores  and  controls  elements  of  the  preliminary  design 
after  evaluation.  In  this  activity,  the  preliminary 
design  and  structure  of  each  CSCI /program/ subsystem  are 
developed,  in  terms  of  computer  software  components 
(CSCs) /modules  or  software  units.  The  software 
engineering  activities  are  supported  by  (a  set  of) 

PDRs.  The  essential  engineering  efforts  and  decisions 
generated  during  this  process  are  normally  documented 
in  preliminary  versions  of  the  SDD,  IDD,  or  US  and  DS. 
Even  though  not  an  element  of  the  configuration 
identification,  the  contracting  activity  approved  STP 
is  also  placed  under  the  contractor's  internal  change 
control.  This  internal  control  of  the  contractor's  STP 
is  necessary  to  ensure  that  the  scope  of  the  software 
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test  program  will  be  implemented  as  mutually  agreed 
between  the  Government  and  the  contractor.  In  addition 
to  the  formal  configuration  baselines  (functional, 
allocated  and  product) ,  the  software  development 
contractor  initiates  internal  control  over  the  evolving 
software  design  and  code,  termed  the  developmental 
configuration,  for  each  item  of  software.  This 
interna!  "baseline”  bridges  the  gap  between  the 
software  portions  of  the  allocated  and  product 
baselines. 


6-14.  Metrics  during  preliminary  design 

The  metrics  with  a  check  mark  apply  at  this  time: 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


/  Reqts  Traceability 
/  Reqts  Stability 
/  Design  Stability 
/  Complexity 


Breadth  of  Testing 
Depth  of  Testing 
Fault  Profiles 
Reliability 


a.  Initial  data  as  a  foundation  for  the  design 
stability  and  complexity  metrics  become  available  at 
this  time.  More  detailed  information  should  be 
available  for  existing  NDI  software  that  is  to  be 
modified. 

b.  The  SEE  metric  applies  as  one  of  the  factors  in 
evaluating  the  capabilities  of  various  software 
contractors. 

c.  Additional  information  on  metrics  as  applied  in 
this  phase  can  be  found  in  Chapter  17,  Software  T&E 
Metrics,  of  this  part  of  DA  Pamphlet  73-1. 


€-15.  PDR  conduct  and  decision  criteria 

Decision  criteria  for  this  phase  are  shown  in  Table  6- 
4. 

a.  Depending  on  the  complexity  of  the  software 
system,  the  PDR  may  be  presented  as  an  interactive 
process  allowing  the  PDR  participants  lead  times  to 
address  large  amounts  of  information.  For  more  complex 
systems,  several  PDRs  may  be  held  in  an  iterative 
fashion  to  allow  design  to  progress.  Regardless  of  the 
method  of  conduct,  a  PDR  should  be  considered 
successfully  complete  only  after  all  responsible 
organizations  have  presented  their  assessment  of 
adequacy,  and  only  if  the  proposed  preliminary  software 
design,  the  proposed  interface  design  document  and  the 
proposed  software  test  plans  are  approved  by  the  PM 
with  concurrence  from  the  PDR  participants. 

b.  The  PDR  needs  to  be  a  realistic  review.  The 
overall  objective  is  to  provide  the  PM  with  enough 
information  to  make  an  informed  decision.  It  is 
imperative  that  information  be  shared  between  the 
software  T&E  community  as  well  as  the  PM.  Active 
participation  in  the  form  of  presentations  by  the  user, 
software  engineering,  and  software  quality  assurance 
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Tabla  6-4.  PraliBinary  Daaign  Phasa 


Primary 


Responsibility 

Products 

Decision  Criteria 

Developer  and  PM 

Develop 

Preliminary 

Design 

Program  design 
language  (PDL)  for 
top  level  design 

User  demonstrations 

PDR(s)  conducted 

S/W  Developer, 

Gov't  SQA  or 

IV6V,  and  users' 
representative 

Requirements  Trace(s) 

Metrics  Report (s) 

Updated 

Acceptable  degrees  of: 
requirements 
traceability  and 
stability;  CRU 
allocations;  design 
stability;  complexity 

(and  others)  is  essential  ii  achieving  the  PDR 
objectives.  These  presentations  should  include  results 
from  the  continuous  evaluations  conducted  to  date.  The 
PM  must  be  presented  with  all  of  the  facts  and 
perspectives  in  order  to  assess: 

(1)  The  compatibility  between  the  allocated 
baselined  software  requirements  and  the  proposed 
preliminary  software  design. 

(2)  The  technical  adequacy  of  the  proposed 
preliminary  software  design  (SDD  Part  I) . 

(3)  The  adequacy  of  the  software  test  plan. 

(4)  Current  metrics  and  other  results  from  the 
continuous  evaluations. 

c.  A  successful  PDR  provides  several  benefits  to 
the  software  T&E  community.  The  major  benefit  is  the 
knowledge  that  the  proposed  preliminary  software  design 
will  adequately  address  the  baselined  software 
requirements.  Additionally,  a  successful  PDR  provides 
the  knowledge  that  the  proposed  software  test  plan  will 
provide  testing  which  will  allow  the  software  T&E 
community  to  fully  evaluate  the  software  system 
performance.  All  in  all,  a  successful  PDR  will  go  a 
long  way  towards  minimizing  program  costs  and  schedule 
risks,  and  in  improving  the  overall  quality  of  the 
design. 

d.  The  results  of  the  requirements  traceability, 
requirements  stability,  design  stability,  and  CRU 
metrics  should  be  presented  at  the  PDR.  The 
requirements  should  show  an  acceptable  degree  of 
completeness,  traceability  (SDD,  software  test  plan 
(STP)  to  SRS/US/DS) ,  and  stability  (SRS,  US,  DS,  IRS, 


6-12 


30  8«pt«mb«r  1992 


DA  Pui  73-1,  Part  8«T«a  (Draft) 


SSS^  SS/FDf  and  UFD) .  CRU  allocations  to  lower  level 
software  should  not  exceed  overall  requirements. 

e.  The  continuous  evaluation  process  provides  the 
PM  with  information  to  make  the  decision  to  move  to  the 
next  phase (s) .  This  warrants  that  the  documents, 
metrics,  reviews,  evaluations,  etc.,  are  in  the 
condition  to  be  used  to  execute  the  next  phases  or  that 
shortfalls  and  risks  have  been  noted. 

8eetioB  VI 

Detailed  Design  Phase 
€-16.  Introduction 

The  detailed  design  phase  can  be  described  simply  as 
the  developer's  efforts  to  allocate  the  baselined 
software  top-level  design  to  the  detailed  software 
design.  Under  normal  circumstances,  the  detailed 
design  is  presented  in  one  or  a  series  of  critical 
design  reviews  (CDRs)  conducted  by  the  software 
developer  upon  completion  of  a  proposed  detailed 
software  design  and  prior  to  the  coding  of  that  design. 

€-17.  Detailed  design  activities 

a.  Objective  one  requires  the  demonstration  that 
the  detailed  software  design  is  adequate  to  permit 
software  code  development.  In  order  to  verify  this 
objective,  the  following  issues  must  be  examined. 

(1)  Traceability  should  be  evaluated  for 
consistency  between  the  approved  preliminary  software 
design  and  the  proposed  detailed  software  design.  Both 
the  US  and  DS  pE  the  SDD  Part  II  and  the  IDD  are 
utilized  to  document  detailed  software  designs. 

(2)  The  detailed  design  should  be  evaluated  to 
determine  the  adequacy  of  the  design  to  perform  the 
functions/components  identified  in  the  preliminary 
design  including  both  internal  and  external  interface 
requirements.  The  detailed  design  should  define  a 
hierarchical  structure  of  identifiable  programs, 
subprograms,  modules  and  units  with  the  highest  level 
of  control  logic  resident  at  the  top  of  the  hierarchy 
and  the  computations  or  arithmetic  functions  resident 
at  the  lower  levels.  Evaluation  of  the  detailed 
software  design  should  address  consistency, 
feasibility,  supportability,  possible  constraints  and 
rationale  for  the  proposed  approach. 

b.  Objective  two  calls  for  the  presentation  of  the 
proposed  software  test  requirements.  The  proposed 
software  test  requirements  are  defined  in  the 

STD  Part  I  or  software  development  test  plan.  The  STD 
or  software  development  test  plan  presented  at  the  CDR 
contains  the  test  cases  necessary  to  perform  formal 
qualification  or  cycle/system  testing  as  defined  in  the 
software  test  plan  or  management  plan  (MP) .  The  STD 
Part  I  should  be  developed  in  accordance  with 
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DOD-STD-2167A,  where  applicable,  and  should  explicitly 
state  software  test  descriptions  and  test  cases  that 
will  be  used  to  test  the  software  including  stress 
testing,  interface  testing,  regression  testing,  and 
built-in  test  validation.  For  information  systems,  the 
CDR  presents  information  necessary  to  determine  if 
planning  for  the  cycle/ system  software  development  test 
(SDT)  has  been  completed,  including  step-by-step 
procedures  to  address  all  test  conditions  lAW  DOD-STD- 
7935A  or  DOD-STD-2167A,  if  applicable. 

c.  Allocation  during  this  phase  is  the  effort  to 
implement  the  top-level  design  down  to  the  explicit 
details  necessary  to  generate  the  end-item  software 
product.  What  results  are  well-defined  details  about 
interfaces,  data  flows,  logic  flows,  databases,  and  all 
other  design  information  short  of  actual  operational 
code.  In  addition,  program  design  language  (PDL)  as 
pseudo-code  is  generated  as  the  link  between  the 
detailed  design  and  end-item  source  code.  Typically, 
if  the  Ada  computer  language  is  used,  the  PDL  will  be 
"compilable”  and  can  actually  be  executed  in  simulators 
to  gain  early,  valuable  information  about  performance. 
This  if  the  opportunity  to  invite  the  users  to  a  user 
demons  ation  to  see  the  design  evolving.  Once  the 
detailed  design  is  deemed  to  be  mature,  a  CDR  is  held 
to  evaluate  and  approve  the  detailed  design  prior  to 
proceeding  to  the  source  code  development  phase.  It  is 
important  to  note  that  full  maturity  is  not  necessary 
to  get  approval  at  CDR,  since  the  developer  will  need 
to  have  the  flexibility  to  refine  and  optimize  the 
design.  However,  the  detailed  design  must  be  mature 
enough  to  allow  for  an  intelligent  decision  to  proceed 
forward  ~  this  is  very  important  both  in  minimizing 
program  cost  and  schedule  risks,  and  in  improving  the 
overall  quality  of  the  design. 

d.  Implementation  of  applicable  metrics. 

e.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  storing  and 
controlling  the  items  of  software  design  as  they  are 
evaluated  and  approved.  Any  changes  to  the  design 
identified  by  the  evaluation  efforts  are  also 
identified  and  processed  by  the  configuration 
management  function  prior  to  implementation.  In  this 
activity,  the  contractor  develops  the  detailed 
structure  or  design  of  each  software  configuration  item 
or  subsystem/program,  in  terms  of  CSCs /modules /units 
and  computer  software  units  (CSUs) .  The  software 
engineering  activities  are  supported  by  (a  set  of) 

CDRs.  The  essential  engineering  efforts  and  decisions 
generated  during  this  activity  are  normally  documented 
in  the  SDD  and  IDD  or  the  US  and  DS.  The  contractor's 
detailed  software  formal  qualification  test  program,  in 
terms  of  test  responsibilities  and  test  cases,  is 
documented  in  one  or  more  STDs  or  test  cases  in  the  PT. 
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The  contractor  places  the  complete  versions  of  the  SDD 
and  the  IDD  or  the  US  and  DS  under  internal  change 
control,  as  part  of  the  developmental  configuration. 
Even  though  not  an  element  of  the  configuration 
identification  or  the  developmental  configuration,  the 
contracting  activity  approved  initial  version  of  the 
STD  or  PT  is  also  placed  under  internal  contractor 
change  control,  after  evaluation  for  adequacy, 
consistency,  and  traceability  by  the  test  and 
evaluation  function. 

6-18.  Metrics  during  detailed  design 

The  metrics  with  a  check  mark  apply  at  this  time: 

/  CoBt  /  Rcqta  Tr»c«ability  Breadth  of  Tasting 

/  Schedule  /  Reqts  Stability  Depth  of  Testing 

^  CRU  /  Design  Stability  Fault  Profiles 

/  SEE  /  Complexity  Reliability 

a.  The  SEE  metric  applies  as  one  of  the  factors  in 
evaluating  the  capabilities  of  various  software 
contractors. 

b.  Additional  information  on  metrics  as  applied  in 
this  phase  can  be  found  in  Chapter  17,  Software  T&E 
Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

6-19.^  CDR  conduct  and  decision  criteria 

Decision  criteria  for  this  phase  are  shown  in  Table  6- 

5. 


Table  6-5.  Detailed  Design  Phase 


Primary 


Responsibility 

Products 

Doclsion  Criteria 

Developer  and  PM 

PDL,  US(S)/S0D, 
STO/SDT,  PT 

PDI.  for  detailed 
design 

User  demonstrations 

CDR(s)  conducted 

PM  fi  Software 
Developer  e 

Gov't  SQA  or 

IVfiV 

Requirements  Trace(s) 

Metrics  Report (s) 

Updated 

Acceptable  degrees  of: 
requirements  traceability 
and  stability;  CRU 
allocations;  design 
stability;  complexity 

a.  For  most  software  systems,  the  CDRs  are  reviews 
which  take  a  substantial  amount  of  time  in  order  to 
adequately  address  all  necessary  issues.  For  some 
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systems,  it  would  be  appropriate  to  allocate  two  to 
three  weeks  for  the  CDRs.  A  successful  approach  used 
in  the  past  has  been  to  present  the  CDRs  in  iterative 
sections  allowing  time  between  each  section  for  CDR 
participants  to  preview  upcoming  CDR  information. 
Regardless  of  the  approach  used  allocating  a 
substantial  amount  of  time  can  greatly  influence  the 
success  of  the  CDRs.  Particular  attention  must  be 
given  to  allowing  all  responsible  organizations  to 
present  their  assessment  of  adequacy.  If  there  is  no 
IVfitV  agent  assigned  to  the  program,  then  the 
developmental  and  operational  evaluators  will  provide 
their  assessments  of  the  progress  of  the  design.  After 
completion  of  the  CDRs,  the  developer  should  publish 
and  distribute  copies  of  the  CDR  minutes. 

b.  The  CDRs  need  to  be  realistic  reviews.  The 
overall  objective  is  to  provide  the  PM  with  enough 
information  to  make  an  informed  decision  to  proceed. 
Active  participation  in  the  form  of  presentations  by 
the  user,  software  engineering,  software  quality 
assurance  (and  others)  are  essential  in  achieving  the 
CDRs  objectives.  These  presentations  should  include 
the  results  from  the  continuous  evaluations  conducted 
to  date.  Again,  if  there  is  no  IV&V  agent,  the 
developmental  and  operational  evaluators  will  provide 
their  assessments  of  progress.  The  PM  must  be 
presented  with  all  of  the  facts  and  perspectives  in 
order  to: 

(1)  Assess  that  the  proposed  detailed  software 
design  is  adequate  to  permit  software  code  development 
(SDD  Part  II  and  IDD) . 

(2)  Assess  the  adequacy  of  proposed  test 
requirements  (STD  Part  I) . 

(3)  Assess  current  metrics  and  other  results 
from  the  continuous  evaluations. 

c.  Successful  completion  of  CDRs  will  provide 
several  benefits  to  the  software  T&E  community.  Among 
these  benefits  is  the  knowledge  that  the  detailed 
design  successfully  addresses  all  system  requirements 
and  is  adequate  to  proceed  with  the  development  of 
software  code,  and  that  no  concerns  from  the  software 
T&E  community  have  been  overlooked.  All  in  all,  a 
successful  CDR  will  go  a  long  way  towards  minimizing 
program  costs  and  schedule  risks,  and  in  improving  the 
overall  quality  of  the  design. 

d.  The  results  of  the  requirements  traceability, 
requirements  stability,  and  CRU  metrics  should  be 
presented  at  the  CDR.  The  requirements  should  show  an 
acceptable  degree  of  completeness,  traceability,  and 
stability.  CRU  allocations  to  lower  levels  of  software 
should  be  compliant  with  the  allocations  in  the 
specifications.  If  complexity  values  for  CSUs  exceed 
thresholdr,  redesign  should  occur  unless  an  adequate 
rationale  is  given. 
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Chaptar  7 

T&E  During  Softvara  Davalopaant  Phasa 
7-1.  Introduction 

chapter  discusses  the  software  activities 
-nvolved  during  the  following  stages  of  software 
development:  coding  and  CSU/unit/module  testing, 
CSC/program  integration  and  testing,  CSCI/cycle/ 
subsystem  testing,  and  system  integration  and  testing. 
It  describes  the  types  of  documents  developed,  tests 
and  evaluations  to  be  conducted,  and  decision  criteria 
for  each  level  of  testing. 

The  test  and  evaluation  activities  described  in 
this  chapter  may  be  repeated  for  iterations  of  the 
design.  Frequently  some  portions  of  the  detailed 
design  are  ready  for  code  and  test  prior  to  others. 

This  provides  the  flexibility  to  move  on  for  software 
which  meets  the  criteria  for  this  phase.  The  objective 
IS  to  provide  flexibility  to  use  a  variety  of 
development  methods  (spiral,  incremental,  sequential). 

c.  During  all  levels  of  software  development 
testing,  the  software  developer  has  control  over  the 
software  TiE  process.  Software  quality  assurance  and 
validation  and  verification  efforts  are  significant 
during  each  level  of  software  development  testing. 

7-2.  Coding  and  CSD/unit/module  taat 
a.  Test  activities. 

4.V  During  coding  and  unit  (CSU) /module  testing, 

folder  (SDF)  or  program  folder 
(PF)  IS  the  key  to  accountability.  Software  is 
developed  in  accordance  with  the  detailed  design  and 
software  coding  standards;  unit  test  procedures  are 
developed  by  the  coder.  Static  analysis,  data  flow 
analysis  and  code  walk-throughs  are  performed  to  assess 
software  modularity,  quality,  and  maintainability  early 
in  the  development.  Use  of  coding  standards  is 

paramount  to  controlling  coding  and  leads  to  a  better 
product • 

(2)  Coding  and  qnit  (CSU) /module  testing  is  the 

lowest  level  of  testing  executed  on  software.  The 
purpose  of  unit  testing  is  two-fold.  First,  to 
validate  the  requirements  expressed  in  the  unit 
specifications  or  software  requirements  specifications. 
Second,  to  ensure  that  all  unit/module  source 
statements  have  been  executed,  each  conditional  branch 
has  been  taken,  and  that  all  boundary  values  (e.g 
minimum-maximum  values)  and  exit  objectives  are 
accepted.  Testing  during  this  phase  is  conducted  bv 
the  developer.  ^ 

(3)  Considerations  for  coding  and  CSU/module 
testing  include  the  test  environment  and  test  data 
The  test  environment  for  module  testing  usually 
consists  of  local  test  bed  hardware.  Test  data  at  this 
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test  level  will  be  generated  by  the  developer;  if  a  set 
of  benchmark  test  files  (BMTF)  exists,  it  should  be 
used  in  addition.  It  is  important  that  results  of 
testing  are  recorded  in  the  SDFs  or  PFs  along  with  the 
inputs  that  generated  those  results.  Only  units  which 
have  been -successfully  tested  are  permitted  to  be 
integrated  into  components  or  programs  for  the  next 
level  of  testing. 

b.  Evaluation  activities.  To  gain  insight  into  the 
software  developer's  progress,  SDFs  or  PFs  are 
reviewed.  All  information  concerning  the  status  of  a 
specific  unit  (CSU)  or  module  is  kept  in  the  SDF  or  PF; 
this  typically  includes: 

(1)  Informal  software  test  descriptions, 
developer  test  results,  informal  coding  and  CSU/module 
notes . 

(2)  Identification  of  other  unit/modules  or 
CSU/CSC(s)  that  Interface  with  this  unit  (CSU) /module. 

(3)  Data  flow  diagrams. 

(4)  Test  results. 

(5)  Software  trouble  reports  (STRs)  or  problem 
reports  (PRs) . 

(6)  Application  and  analysis  of  applicable 
metrics. 

(7)  The  requirements  traceability  matrix. 

c.  The  developer  is  responsible  for  the  SDFs  and 
their  contents.  Evaluations  of  the  SDFs  are  usually 
conducted  by  an  independent  organization  or  management 
structure  separate  from  the  software  developer  (i.e., 
developer's  quality  assurance  group  and/or  the 
Government  I s  SQA,  verification  and  validation  (ViV)  or 
IV&V  organizations) .  Completed  and  published  system 
documentation  and  training  packages  are  not  nozmally 
available  at  unit/module  level  testing.  However,  if 
documentation  is  available,  it  should  also  be  reviewed. 

d.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  storing  and 
controlling  the  items  of  software  design  and  code  as 
they  are  tested,  evaluated  and  approved.  Any  changes 
to  the  design  or  code  identified  by  the  test  and 
evaluation  efforts  are  also  identified  and  processed  by 
the  configuration  management  function  prior  to 
implementation.  In  this  activity,  the  code  is 
developed  that  implements  the  previously  defined 
requirements  and  design.  Testing  is  performed  on  the 
actual  coded  items.  The  result  of  this  activity  is  a 
set  of  coded  items,  that  have  been  individually  tested 
and  found  suitable  for  integration  into  the  completed 
software  product.  The  configuration  management 
function  supports  the  test  and  evaluation  efforts  by 
storing  and  controlling  each  item  as  it  is  coded  and 
evaluated.  The  contractor /developer  controls  changes 
to  the  detailed  design  that  result  from  the  code  and 
unit  test  activities.  As  each  software  unit 
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successfully  passes  unit  test,  it  is  entered  into  the 
developmental  configuration. 

e.  Metrics.  The  metrics  with  a  check  mark  apply  at 
this  time: 


/  Reqts  Tracaability  /  Breadth  of  Teating 

/  Reqts  Stability  /  Depth  of  Testing 

✓  Design  Stability  /  Fault  Profiles 

/  Complexity  Reliability 

(1)  Measured  values  for  computer  resource 
utilization  become  available  at  this  time. 

(2)  With  the  initiation  of  code  testing,  data  to 
support  the  breadth  and  depth  of  testing  metrics  can  be 
collected  and  analyzed. 

(3)  Additional  information  on  the  application 
and  analysis  of  metrics  can  be  found  in  Chapter  17, 
Software  TiE  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

f.  Decision  criteria.  Associated  products, 
documents  and  decision  criteria  are  shown  in  Tables  7-1 
and  7-2. 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


Table  7-1.  Coding  and  Unit  (CSU) /Module  Test 


Prisisry 


Responsibility 

Products 

Decision  Criteria 

S/W  Developer 
and  Gov't. 

Requirements  Traee(B) 

Updated 

SQA  or  IV&V 

Hetrics  Report (s) 

Acceptable  degrees  of: 
requirements 
traceability  and 
stability;  CRU 
allocations  to 
utilization;  design 
stability;  breadth  and 
depth  of  testing; 
fault  profiles 

Table  7-2.  Coding  and  Unit  (C8U) /Module  Teat  Documents 


Primary 

Responsibility 

N8CR  1 

Products 

Condition 
of  Document 

AZ8  Products 

Decision 

Criteria 

Software 

Developer 

CRISD 

Preliminary 

Maintenance 

Manual 

Draft 

a 

SDFs 

In  Process 

Program  Folders 

None 

STDs 

Draft 

Procedures  for 

SDT  (section  5) 

Draft 
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7-3.  C8C/prograa  intagration  and  taat 

During  this  level  of  testing,  the  developer  integrates 
the  modules  into  programs  or  the  computer  software 
units  into  CSCs.  Unit  level  or  module  level  testing 
must  have  been  successfully  completed  prior  to 
CSC/program  level  testing.  All  CSCs/programs  should 
accept  valid  inputs  and  produce  correct  outputs. 

a.  Test  activities.  Program-level  software 
development  plan  or  CSC  integration  procedures  will  be 
used  by  the  developer  to  execute  program-level  tests. 

At  this  point,  with  much  of  the  software  integrated, 
limits  and  bounds  are  tested,  and  multiple  paths  are 
executed  to  ensure  that  the  integration  is  proceeding 
in  a  robust  manner.  Additionally,  testing  such  as  run¬ 
time  efficiency  and  stress  testing  should  also  be 
addressed.  All  discrepancies,  malfunctions  and  errors 
(prioritized  and  categorized  lAW  DOD-STD-2167A, 

Appendix  C)  should  be  documented  in  problem  reports  or 
trouble  reports.  Results  of  the  testing  are  recorded 
in  SDFs  or  PFs. 

b.  Evaluation  activities.  Software  quality 
assurance,  IV&V,  and  v&v  personnel  are  generally 
performing  the  c  ;luation  on-site  and  reporting  to  the 
remainder  of  the:  software  TiE  community.  Required 
evaluations  at  this  time  include: 

(1)  Tracing  requirements  from  test  results  back 
to  the  test  plans/procedures  and  the  requirements 
documents . 

(2)  Auditing  SDFs /PFs. 

(3)  Implementation  and  analysis  of  applicable 
metrics. 

c.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  controlling  the 
coded  items  as  they  are  integrated.  Any  changes  to  the 
design  or  code  identified  by  the  test  and  evaluation 
efforts  are  also  identified  and  processed  by  the 
configuration  mamagement  function  prior  to 
implementation.  In  this  activity,  the  coded  items  are 
integrated  into  a  set  of  CSCs  or  programs.  The  result 
of  this  activity  is  a  coded  item  that  is  ready  for 
formal  qualification  testing  or  subsystem/cycle 
testing.  The  detailed  test  procedures  to  be  used  for 
formal  qualification  or  cycle/system  testing  are 
documented  in  the  STD  or  PT  test  cases.  This  activity 
is  supported  by  one  or  more  TRRs  that  determine  the 
readiness  of  the  software  for  formal  qualification  test 
(CSCI  or  cycle  and  system) .  The  developer  controls 
changes  to  the  detailed  design  that  result  from  the 
CSC/program  integration  and  test  activities.  All 
changes  to  the  units/modules  that  result  from 
CSC/program  integration  and  test  are  internally 
controlled  and  entered  into  the  development 

conf iguration . 
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d.  Metrics.  The  metrics  with  a  check  mark  apply  at 
this  time: 


/  Cost 
/  Schedule 
✓  CRU 
/  SEE 


/  Reqts  Traceability 
/  Reqts  Stability 
/  Design  Stability 
/  Complexity 


/  Breadth  of  Testing 
/  Depth  of  Testing 
/  Fault  Profiles 
Reliability 


(1)  Breadth  and  depth  of  testing  become  more 
refined  in  this  phase. 

(2)  Additional  information  on  the  application 
and  analysis  of  metrics  can  be  found  in  Chapter  17, 
Software  TiE  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

e.  Decision  criteria.  Associated  products, 
documents  and  decision  criteria  are  shown  in  Tables  7-3 
and  7-4. 


Table  7-3.  CSC/Program  Integration  and  Test 


Primary 


Raaponsibility 

Products 

Deeisiea  Criteria 

S/W  Daveloper 
and  Gov't. 

Requirements  Trsce(s) 

Updated 

SQA  or  IV&V 

Metrics  Report (8) 

Acceptable  degrees  of: 
requirements 
traceability  and 
stability;  CRU 
allocations  to 
utilization;  design 
stability;  breadth  and 
depth  of  testing; 
fault  profiles 

PM  &  Davelopar 

TRR(S) 

Ready  to  perform  CSCI 

with  SQA  and  IV&V 

&  system/cycle  tests 

7-4.  CSCI/eycle/subsystem  teat 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  CSCI/cycle/subsystem  level 
tests.  During  this  level  of  testing,  the  developer 
integrates  programs  into  subsystems  or  the  CSCs  into 
CSCIs.  Program  or  CSC  level  testing  must  have  been 
successfully  completed  prior  to  CSCI/ subsystem  level 
testing.  It  is  sometimes  combined  with  system  level 
testing  since  interfaces  need  to  be  exercised.  If 
combined,  all  the  test  requirements  and  products  shown 
in  both  test  descriptions  apply.  Prior  to  running  each 
subsystem/CSCI /cycle  test,  a  formal  test  readiness 
review  lAW  MIL-STD-1521B  covering  that  test  must  have 
been  held  by  the  PM  and  developer. 

b.  Test  activities. 

(1)  Formal  developer's  tests  consist  of 
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Tabla  7-4.  CSC/Progran  Intagration  Docunants 


Primary 

Rasponaibility 

MSCR 

Products 

Condition 
of  Docuaont 

AZ8  Products 

Decision 

Criteria 

Software 

Developer 

CRISD 

Preliminary 

Maintenance 

Manual 

Draft 

SDFs 

In  Process 

Program  Folders 

None 

STDs 

Draft 

Step-by-step 
procedures  in  PT 

Draft 

User  Manual  (UM) 

Draft 

End  User  Manual 
(EM) 

Draft 

CSC I /subsystem  or  cycle/system  levels  of  testing.  This 
testing  involves  the  combining  or  linking  of 
modules /programs  or  CSCI/ subsystems  into  major 
processes  (e.g.,  basic  cycle,  as-required  cycle,  fire 
control,  missile  guidance).  Considerations  for  this 
testing  level  include:  software,  software  development, 
test  plan,  the  test  environment,  test  data, 
documentation,  training  materials  and  evaluation  of  the 
test.  The  test  plan  will  be  prepared  by  the  tester. 

The  test  plans  need  to  consider  the  operational  mode 
summary/mission  profile  from  the  UFD  in  planning  peak 
loads  and  volumes,  numbers  of  transactions,  and 
multithread  test  conditions. 

(2)  The  CSCI /subsystem  test  plan  (STP)  and 
associated  software  test  descriptions  (STOs)  are 
reviewed  and  approved  by  the  Government;  use  of  a 
quality  assurance  group  is  required.  Cycle/system  or 
CSCI /subsystem  level  testing  will  be  executed  using 
local  test  bed  hardware,  preferably  on  the  target 
hardware.  Test  drivers  for  the  CSCI  or  subsystem  test 
which  emulate  system  loading  consistent  with  the  OMS/MP 
are  essential  for  early  identification  of  faults  which 
impact  the  accomplishment  of  user  requirements.  BMTF 
may  also  be  used  for  this  level  of  testing.  Draft 
software  documentation  materials  being  prepared  by  the 
software  developer  must  continue  to  be  developed  and 
refined  so  that  the  final  developer  test  iterations  of 
testing  will  use  final  draft  versions  of  the 
documentation  items. 

(3)  As  part  of  the  test  reporting  process,  all 
problems  and  deviations  encountered  during  the  tests 
will  be  prioritized,  recorded,  and  placed  into  the  SDFs 
or  program  folders  and  reviewed  by  the  appropriate 
functional  or  technical  representative.  The  T6E 
community  will  assist  in  prioritizing  the  PRs  or  STRs. 
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c.  Evaluation  activities. 

(1)  Cycle  tests  conclude  with  the  tester's 
preparation  of  a  draft  test  analysis  report. 

(2)  The  final  report  includes  system  level 
testing.  Only  when  finalized  does  it  get  furnished  to 
the  PM  and.  others.  The  developer  concludes 

CSCI/ subsystem  test  activities  by  updating  the  source 
code  and  design  documentation,  recording  the  STRs  or 
PRs,  and  preparing  the  test  report.  It  is  furnished  to 
the  user  representative  and  PM/PO  for  comments.  After 
consideration  of  comments,  it  will  be  forwarded  for 
review  by  the  approving  authority  and  the  developmental 
tester,  developmental  evaluator,  operational  evaluator, 
and  operational  tester. 

(3)  CSCI /subsystem  tests  are  generally  the 
officially  witnessed  software  acceptance  tests  (formal 
qualification  tests  (FQTs)),  with  official  Government 
representatives  present  for  the  test  execution. 

(4)  The  implementation  and  analysis  of  the 
applicable  metrics  are  part  of  CE. 

d.  The  configuration  management  function  supports 
the  test  and  evaluation  efforts  by  ensuring  that  the 
correct  versions  of  the  coded  items  are  provided  for 
the  formal  qualification  testing,  and  that  only 
approved  changes  are  incorporated  into  the  tested 
items.  In  this  activity,  formal  qualification  testing 
of  each  CSCI  is  conducted  according  to  the  STP  and 
STDs,  and  documented  in  (a  set  of)  STRs  or  RTs. 
Following  successful  completion  of  this  testing  or 
system  integration  testing,  an  FCA  and  PCA  are 
performed  for  each  software  Cl.  The  purpose  of  the  FCA 
is  to  determine  that  the  software  performs  in 
accordance  with  the  requirements  documented  in  the 
current  configuration  identification.  The  purpose  of 
the  PCA  is  to  determine  that  the  coded  software  is 
accurately  and  completely  reflected  in  its  associated 
documentation,  such  as  the  software  product 
specification  (SPS)  which  includes  the  software 
engineering  environment  (SEE) ,  the  IDD,  or  the 
completed  US,  DS,  SSS,  and  the  source  and  object  code 
listings.  The  contractor  identifies  the  exact  version 
of  each  software  item  to  be  delivered  in  a  VDD  or 
implementation  procedures  (IP) .  Following  completion 
of  the  preliminary  FCA  and  PCA,  the  Army 

Program/ Project  Office  authenticates  the  SPS  or  the 
design  and  build  documents  and  enters  them  into  the 
product  baseline  for  the  system.  The  contractor 
ensures  that  all  changes  that  result  from  testing  are 
internally  controlled  and  entered  into  the 
developmental  configuration.  Once  the  software  is 
entered  into  the  product  baseline,  all  changes  to  the 
software  require  prior  approval.  At  this  point,  the 
developmental  configuration  ceases  to  exist.  It  is 
recommended  that  a  preliminary  or  iterative  PCA  and  FCA 
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be  done  and  updated  during  the  system  test  phase. 

Final  results  should  be  an  input  to  the  maintenance 
evaluation. 

e.  Metrics.  The  metrics  with  a  check  mark  apply  at 
this  time: 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


/  Reqts  Traceability 
/  Reqts  Stability 
/  Design  Stability 
/  Complexity 


/  Breadth  of  Testing 
/  Depth  of  Testing 
/  Fault  Profiles 
/  Reliability 


f.  Decision  criteria.  Associated  products, 
documents  and  decision  criteria  are  shown  in  Tables  7-5 
and  7-6. 


Table  7-5.  C8Cl/Cycle/8ubsystem  Test 


Primary 


Responsibility 

Products 

Decision  Criteria 

S/W  Developer 
and  Gov't. 

Requirements  Trace ( s ) 

Updated 

SQA  or  IV&V 

Metrics  Report (s) 

Acceptable  degrr  of: 

requirements 
traceability  and 
stability;  CRU 
allocations  to 
utilization;  design 
stability;  breadth  and 
depth  of  testing; 
fault  profiles; 
reliability 

7-5.  8ystem  integration  test 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  system  integration  tests. 
During  this  level  of  testing,  the  developer  integrates 
CSCIs  with  CSCIs  and  hardware/ software  configuration 
items  to  complete  the  system.  Cycle  or  CSCI  testing 
must  have  been  successfully  completed  prior  to  system 
testing.  As  indicated,  it  may  be  combined  with  CSCI 
and  cycle  testing. 

b.  Test  activities. 

(1)  Formal  developer's  tests  consist  of  system 
integration  testing.  This  testing  involves  the 
combining  or  linking  of  CSCI  or  subsystems  into  major 
processes.  Considerations  and  responsibilities  for 
this  testing  level  include:  software,  software 
development,  test  plan,  the  test  environment,  test 
data,  documentation  and  training  materials  and 
evaluation  of  the  test.  The  test  plan  will  be  prepared 
by  the  tester  (e.g. ,  developer)  and  will  use  the  loads, 
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Tabla  7-6.  CSCI/Cyela/Subsystam  Documants 


Primary 

Raiponaibility 

NSCR 

Products 

Condition 
of  Ooeuaont 

AXS  Products 

Decision 

Criteria 

Software 

Developer 

CRISD 

Preliminary 

Maintenance 

Manual 

Final 

SDFa 

In  Process 

Program  Folders 

None 

STDs 

Final 

User  Manual  (UM) 

Draft 

STHs 

In  Process 

PRs 

None 

Software 

Product 

Spec. 

(SPS) 

Draft 

Computer 

Operation 

Manual 

En'd  User 

Manual  (EM) 

Draft 

Draft 

Ver. 

Descrip. 

Document 

Draft 

Implementation 

Procedures 

Draft 

s/w 

Test 

Report 

Final 

Test  Reports 

Draft  - 

Cycle 

Portion 

volumes,  stresses,  and  conditions  documented  in  the 
users'  functional  descriptions  as  the  basis  for  test 
design. 

(2)  The  system  integration  test  plan  and 
associated  STDs  are  reviewed  and  approved  by  the 
Government.  The  cycle/system  test  plan  is  reviewed  and 
approved  by  the  Government  (if  contracted  out) .  For 
all  software,  system  integration  testing  will  be 
executed  using  local  test  bed  hardware,  preferably  on 
the  target  hardware.  BMTF  may  be  used  for  this  level 
of  testing.  Draft  software  documentation  materials 
being  prepared  by  the  software  developer  must  continue 
to  be  developed  and  refined  so  that  the  final  developer 
test  iterations  will  use  final  draft  versions  of  the 
documentation  items . 

(3)  As  part  of  the  test  reporting  process,  all 
problems  and  deviations  encountered  during  the  tests 
will  be  prioritized,  recorded,  and  placed  into  the  SDFs 
or  program  folders  and  reviewed  by  the  appropriate 
functional  or  technical  representative.  The  T&E 
community  will  assist  in  prioritizing  the  PRs  or  STRs 
through  the  software  corrective  action  review  team 
(SCART) . 

c.  Evaluation  activities. 

(1)  System  integration  testing  concludes  with 
the  tester's  preparation  of  a  test  report.  It  is 
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furnished  to  the  user  representative  and  PM  and 
conmented  upon.  After  consideration  of  comments,  the 
software  test  report  will  be  forwarded  for  review  by 
the  developmental  tester,  developmental  evaluator, 
operational  evaluator,  and  operational  tester.  The 
software  test  report  forms  the  basis  for  the  PM's 
developmental  test  readiness  statement  (DTPS)  for 
software  related  issues. 

(2)  The  developer  concludes  system  integration 
test  activities  by  updating  the  source  code  and  design 
documentation,  resolving  the  STRs  or  PRs  and  preparing 
the  test  report.  System  tests  may  be  officially 
witnessed  software  acceptance  tests  with  official 
Government  representatives  present  for  the  test 
execution.  Metric  implementation  during  the  system 
testing  phase  is  dependent  on  the  number  and  severity 
of  errors  detected  and  their  effect  upon  the  software 
requirements  and  design.  Corrective  action  may  require 
requirements  changes,  redesign,  and  regression  testing 
at  all  levels  (i.e.,  revalidation) . 

(3)  Functional  and  physical  configuration 
audit (s)  lAW  DOD-STD-1521B  as  a  minimum,  and  a 
maintainability  evaluatior  will  be  performed  to  ensure 
that  the  software  documentation  and  performance  meets 
requirements.  The  maintainability  evaluation  will  be 
performed  by  PDSS  personnel  to  demonstrate  that  the 
software  can  be  maintained  lAW  the  approved  maintenance 
concept  and  provide  input  to  the  required  software 
supportability  and  suitability  statement  for  materiel 
release.  The  approved  maintenance  concept  indicates 
whether  the  software  will  be  maintained  by  a  designated 
Government  CDA  or  LCSEC  or  whether  it  is  contractor 
logistics  support  (CLS) .  Issues  addressed  in  the 
evaluation  may  include  the  items  below  ensuring  that: 

(a)  Sufficient  resources  exist  for  support, 

including  additional  computers,  validated  tools,  code 
generation  tools  such  as  compilers,  linJcers,  and 
debuggers ;  requirements  and  design  tools  such  as 
computer  aided  software  engineering  (CffiSE)-  tools;  such 
as  docxmentation  and  training.  ' 

(b)  Software  docvunentation  is  understandable, 
complete,  and  in  a  format  that  is  compatible  with  the 
software  tools  being  used. 

(c)  Complexity  metrics  are  reviewed  for  code 
simplicity. 

(d)  Programmer's  manuals  are  adequate  and 
programmers  are  trained  in  all  required  languages. 

(e)  Computer  resource  utilization  (CRU) 
metrics  are  reviewed  to  ensure  that  sufficient 
memory  and  timing  reserves  exist  to  allow  for  software 
maintenance. 

(f)  Transition  plans  for  software  support  are 

reviewed. 

(g)  The  results  from  FCA  and  PCA  are 
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incorporated  into  the  production  baseline. 

(h)  A  configuration  control  board  has  been 
established. 

(i)  Configuration  management  policies  and 
procedures  are  in  place. 

tj)  Configuration  control  mechanisms  for 
release  and  distribution  are  in  place. 

(k)  The  status  of  all  metrics  collected  are 
reviewed  to  determine  requirements  met,  depth  and 
breadth  of  testing,  and  faults  remaining. 

d.  Metrics.  The  metrics  with  a  check  mark  apply  at 
this  time: 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


/  Reqts  Traceability 
/  Reqts  Stability 
/  Design  Stability 
/  Complexity 


/  Breadth  of  Testing 
/  Depth  of  Testing 
/  Fault  Profiles 
/  Reliability 


e.  Decision  criteria.  Associated  products, 
documents  and  decision  criteria  are  shown  in  Tables  7-7 
and  7-8. 


Table  7-7.  System  Test 


Primary 


Responsibility 

Products 

Decision  Criteria 

S/W  Developer 
and  Gov't. 

Requirements  Trace(s) 

Updated 

SQA  or  IV&V 

Metrics  Report (s) 

Acceptable  degrees  of: 
requirements 
traceability  and 
stability;  CRU 
allocations  to 

utilization;  design 
stability;  breadth  and 
depth  of  testing; 
fault  profiles; 
reliability 
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Tabla  7-8.  8yataB  Tast  DoeuBants 


Primary 

Rasponsibility 

MSCR 

Products 

Condition 
of  Document 

AIS  Products 

Decision 

Criteria 

Software 

Developer 

CRISO 

Pinal 

Maintenance 

Manual 

Final 

SDFs 

In  Procast 

Program  Folders 

None 

STDs 

Final 

User  Manual  (UM) 

Draft 

1 

End  User 

Manual  (EM) 

Draft 

STRs 

In  Process 

PRs 

None 

SPS 

Final 

Computer  Operation 
Manual 

Final 

Ver 

Descrip. 

Document 

Final 

j 

Implementation 

Procedures 

Final 

S/W 

Test 

Report 

Final 

Test  Report 

Final 
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Chaptar  8 

8ystaB  Davalopmantai  Tast  and  Evaluation  (DT/DT&E) 

8-1.  Introduction 

a.  This  chapter  describes  the  software  test  and 
evaluation,  considerations  for  systems  containing 
software.  During  this  phase,  the  Government  controls 
the  T&E  process.  The  developmental  test  phase  consists 
of  a  series  of  system-level  tests,  as  defined  in  AR  73- 
1  and  Part  Four  of  DA  Pamphlet  73-1,  not  necessarily 
conducted  consecutively.  The  software  baseline  is 
frozen  for  this  phase  of  testing  in  order  to  maintain 
data  consistency.  The  baseline  is  not  modified  during 
this  phase  of  testing,  unless  severe  problems  are 
encountered.  If  the  baseline  is  modified,  regression 
testing  is  required  to  ensure  detected  problems  were 
corrected  and  additional  problems  were  not  introduced 
into  the  software.  Prior  to  entering  developmental 
test,  a  developmental  test  readiness  review  (DTRR)  is 
conducted  which  determines,  the  maturity  of  the  software 
and  system. 

b.  The  DTRR  convenes  as  a  working  group  whose 
members  include  the  principal  TIWG  members  plus  others 
deemed  appropriate.  As  a  minimtim  a  DTRR  will  be 
conducted  before  developmental  test.  The  DTRR  reviews 
all  pre-start  activities  and  requirements  which  may 
impact  the  execution  of  the  test  as  scheduled  and  as 
planned  by  the  TIWG.  The  objective  of  the  review  is  to 
determine  what  actions  are  required  to  assure  that 
resources,  training,  planning,  and  test  equipment  will 
be  in  place  to  support  the  successful  performance  of 
the  test  and  to  ensure  that  the  T&E  planning, 
documentation,  resources,  design  maturity/ 
configuration,  and  data  systems  have  been  adequately 
addressed.  Principal  DTRR  attendees  include 
representatives  of  the  PM/MATDEV  (Chair)  and  the 
materiel  developer's  safety  office  (if  applicable),  ILS 
office,  and  product  or  quality  assurance  &  testing 
office;  developmental  tester;  developmental 
evaluator /assessor;  operational  tester;  operational 
evaluator;  logistician;  CBTDEV,  trainer,  and  manpower 
and  personnel  integration  (MANPRINT)  discipline 
members.  The  DTRR  conduct  and  other  details  are 
described  further  in  Part  Four  of  DA  Pamphlet  73-1. 
Figure  8-1  shows  a  checklist  of  software  related 
questions  discussed  at  the  DTRR. 

8-2.  Entry  objectives 

a.  In  order  to  enter  developmental  test,  the 
following  must  be  completed:  evidence  of  successful 
completion  of  developer's  tests,  formal  and  informal, 
consisting  of  test  report,  developer  test  documentation 
(e.g.,  updated  test  plan  and  procedures,  STR/PR/test 
incident  reports  (TIR)  or  engineering  change  proposal- 
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1 .  0«n*ral . 

a.  Have  configuration  items  related  to  software  been  identified? 
Are  changes  under  configuration  control? 

b.  Has  government  reviewed /approved  all  software  test 
plans/procedures/test  results? 

c.  Are  all  functional  requirements  clearly  identified? 

d.  Is  there  confidence  that  software  functions  will  execute 
properly  (walk-throughs,  B5/CS  MIL-STD-490,  DOD-STD-2167A  or  DOD-STD- 
7935A  requirements  and  design  specifications,  resource  allocation)? 

e.  Is  there  a  clear  understanding  of  what  software  functions  will 
be  tested  by  the  developmental  and  operational  testers? 

f.  Is  the  Computer  Resource  Management  Plan  current,  if 
applicable?  (see  AMC-R  70-16) 

g.  Have  plans  been  formulated  to  deliver  all  software 
documentation  prior  to  DT/OT? 

2.  Safety.  Does  the  system  or  software  have  any  safety  limitations 
(operational  limitations  for  test  personnel)  either  inside  or  outside 
the  required  performance  envelope?  If  so,  what  corrective  action  has 
been  taken  or  is  any  planned? 

3.  Reliability,  Availability,  Maintainability.  Have  the  failure 
definition/scoring  criteria  for  software  been  established? 

4.  Configuration  Nanagoment  (AR  70-37). 

a.  Has  a  preliminary  product  baseline  technical  document8££bn 
package  been  established? 

b.  How  is  the  software  configuration  being  controlled? 

(1)  Has  a  configuration  management  plan  been  approved  wh.ch 
includes  provisions  for  government  approval  of  engineering  change 
proposals-software  and  waivers /deviations? 

(2)  Has  a  Configuration  Control  Board  been  established? 

5.  Testing.  Compare  the  requirements  document  against  test  results 
to  date.  There  must  be  a  reasonable  assurance  (confidence)  that  the 
system  to  be  tested  can  satisfactorily  pass  technical  tests  or 
equivalent  independent  government  tests. 

a.  Do  test  results  show  that  system  and  software  requirements 
will  be  met?  (Show  quantity  tested,  what  tests  were  conducted, fend 
results). 

b.  Have  the  tests  addressed  all  system  and  software  requirements? 

c.  What  problems  have  occurred  and  how  have  they  been  resolved? 

d.  Have  critical/major  test  incident  reports  from  developmental 
and  operational  testing  been  closed  out?  (List  and  suamtarize 
corrective  action. ) 

6.  Integrated  Logistics  Support. 

a.  Supportability. 

(1)  Has  the  PDSS  agent  been  identified? 

(2)  Has  the  software  maintainability  evaluation  been 

conducted?  .  -^vanct 

b.  Training.  What  version  of  software  was  used  for -l^aining? 

7.  Test  Resources.  Are  any  unique  facilities,  equipment  dr  software 
instrumentation  required  and  will  they  be  available  at  the  test 
8ite(s)? 


Figure  8-1.  DTRR  8oftvare  TtE  Checlclist 
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•software  (ECP-S)  generated) ,  QA  certified  software 
baseline,  baseline  name  and  version  identifiers.  A 
DTRS  is  prepared  by  the  PM  and  submitted  to  the 
developmental  tester.  Preferably,  there  should  be  no 
open  priority  1  or  2  problem  reports  or  TIRs  from 
previous  testing.  Serious  consideration  and  severe 
test  limitations  may  result  if  DT  occurs  with  open 
problem  reports.  It  may  cause  DT  to  be  repeated  due  to 
invalid  data. 

b.  Documents  include  an  updated  ZEP  or  ZAP  if 
independently  evaluated  or  assessed,  either  test  design 
plan  (TDP)  and  detailed  test  plan  (DTP) ,  or  test  plan 
(PT) .  The  system  documentation  for  the  change  pac)cage 
will  be  in  final  draft  form.  An  approved  TEMP  is 
required  to  enter  developmental  testing. 

8-3.  Activities 

a.  System  test.  System  testing  is  generally 
defined  as  testing  which  examines  the  compliance  of  the 
system  (hardware  and  software)  to  its  requirements. 

This  differs  significantly  from  lower  levels  of  testing 
in  which  the  product  is  evaluated  against  its  design. 
Using  the  requirements  as  the  standard  measure,  system 
testing  leads  to  an  evaluation  of  design  rather  than 
evaluation  of  the  implementation  of  the  design.  The 
approach  to  system  level  testing  primarily  utilizes 
blac)c-box  techniques  (i.e.,  examination  of  the 
relationship  of  inputs  to  outputs  as  specified  in  the 
requirements) . 

b.  Formal  developmental  test.  DT  is  a  Government 
system-level  test  which  focuses  on  requirements.  Zt 
verifies  system  performance  and  the  ability  of  the  user 
to  perform  mission-essential  activities.  DT  is 
executed  on  target  hardware  using  conditional  data 
files.  Critical  to  the  success  of  future  tests  is  the 
ability  to  drive  or  load  the  system  software  ZAW  the 
OMS/MP  defined  by  the  user. 

(1)  The  tester  will  coordinate  all  activities 
with  the  PM  and  provide  guidance  for  the  resources 
required  to  support  testing.  The  test  may  be  executed 
using  an  ad-hoc  group  of  system  users  and/or  operators. 
At  the  beginning  of  DT  activities,  computer  operation 
and  users  manuals,  conversion  documentation  and 
training  pac}cage  materials  will  be  in  final  draft  form 
and  will  be  used  in  the  execution  of  the  DT  processes. 

(2)  The  tester  writes  a  DTP  or  PT,  in  response 
to  the  ZEP/ZAP  and  TDP.  Well-defined  requirements  are 
necessary  to  develop  the  DTP  or  PT  for  all  evaluation 
issues.  These  requirements  will  allow  testers  to 
tailor  the  generic  test  areas  to  the  specific  software 
application.  The  following  approach  addresses  the 
software  requirements  in  the  context  of  DT: 

(a)  Trace  critical  system  requirements  to  the 
software  requirements. 
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(b)  Identify  the  input  conditions  possible 
for  each  software  requirement.  This  leads  to  the 
development  of  a  test  coverage  matrix.  The  test 
coverage  matrix  has  cells  which  represent  input 
conditions  and  the  tests  addressing  them. 

Cc)  Design  and  execute  tests  to  address  the 
most  important  test  conditions.  It  will  generally  not 
be  possible  to  provide  coverage  for  all  input 
conditions,  so  testing  should  focus  on  the  more 
critical  or  more  likely  test  conditions  with  respect  to 
the  expected  deployment  environment.  This  serves  not 
only  to  ensure  that  technical  requirements  are  met,  but 
also  ensures  readiness  for  operational  testing 
following  developmental  test. 

(3)  Areas  addressed  in  the  DTP  or  PT  should 
include:  performance,  interoperability,  usability, 
security,  and  safety. 

(4)  The  following  testing  areas  are  addressed 
when  testing  software  performance: 

(a)  Accuracy  —  this  has  the  following 

aspects: 

-  Determine  if  correct  decisions  are  made 
(i.e.,  yes/no,  true/false),  or  the  best 
choice  of  several  alternatives  is  chosen. 

-  Proximity  to  expected  values.  This  is 
applicable  when  a  numerical  computation  is 
performed  and  there  is  a  tolerance 
associated  with  resultant  output.  A 
computation  is  considered  accurate  when  it 
is  within  a  specified  tolerance. 

(b)  Speed  —  determine  if  functions  are 
performed  within  specified  time  tolerances. 

(c)  Repeatability  —  detexrmine  if  consistent 
conditions/events  produce  consistent  results  or  if  the 
software  appears  to  compute  in  a  random  way. 

(d)  Robustness  —  by  this  we  mean  "crash- 
resistance."  This  applies  to  any  area  of  testing 
whether  computational,  interoperative  or  operational. 

(e)  Stress  —  verify  that  volumes,  loads, 
varying  conditions,  or  peak  processing  do  not  degrade 
the  system  except  lAW  requirements . 

(5)  Testers  address  the  two  interoperability 
areas,  inter system  and  intrasystem,  in  the  same  way. 

The  following  are  checked: 

(a)  Acceptance  of  legal  transmissions. 

(b)  Rejection  of  illegal  transmissions.  The 
system  should  not  degrade  or  crash  when  illegal 
attempts  are  made. 

(6)  Usability  performance  testing  addresses  the 
user-machine  interface.  Test  areas  addressed  are: 

(a)  Acceptance  of  legal  input. 

(b)  Rejection  of  illegal  input.  Again,  the 
system  should  reject  illegal  values,  not  degrade  or 
crash. 
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(c)  User  understandability.  Determine  if 
displays/entries  are  understandable  for  the  user. 

(d)  Determine  if  correct  input  yields 
expected  output. 

testing  is  required  for  software. 

Jho  perforaed  if  the  software  could  cause 

the  system  to  engage  in  an  unsafe  action  toward 
personnel,  equipiaent  or  materiel. 

(8)  The  independent  developmental  evaluator 

but  not  participate  as  a  test 
independent  developmental  assessor  may 
actually  be  the  test  director. 

developer  is  not  to  participate  as  a 

member.  When  difficulties  occur, 
the  deyelopmental  tester  may  request  the  developer's 

policy  is  to  control  changes  to  the 
software  during  the  testing,  if  the  numbers  of 
problems  reported  at  the  priority  1,  2,  or  3  level 
becpome  excessive,  thereby  impairing  the  test 
objectives,  the  developmental  tester  will  suspend  or 
terminate  the  testing  in  coordination  with  the 
de ve lopmenta 1  eva luator/ assessor . 

(10)  Problems  discovered  during  DT  are  recorded 
®y®'t®m/software  anomalies.  Problems  reported  use 
either  the  TIR,  PR  or  similar  format.  These  forms  have 
a  developer's  analysis  section  which  is  used  for 
corrective  action  reporting.  All  software  problems 
occurring  during  test  (including  problems  with  test 
procedures)  will  be  recorded,  categorized  and 
PJ^ioritized  lAW  DOD-STD-2167A,  Appendix  C. 

.4  w  contributing  toward  reliability,  depth 

and  breadth  of  testing,  and  fault  profiles  is  ^ 

collected.  This  loo)cs  at  software  faults  discovered 
and  closed,  test  adequacy,  and  failure  patterns, 
c.  Evaluation. 

.  Results  of  testing  are  prioritized  and 

categorized  by  a  Government  data  review  board.  Board 
principal  members  shall  be  the  PM,  user  representative, 
developmental  evaluator /assessor,  and  developmental 
tester.  Test  results  are  evaluated  by  the 
developmental  evaluator/assessor  lAW  the  lEP/IAP 

(2)  Developmental  evaluation  is  a  comprehensive 
and  validation  process  conducted  to  ensure 
that  all  capabilities  and  requirements  of  the  system 

®f®^®^®^®^®®‘^  analyzed  lAW  the  issues  and  criteria 
stated  in  the  lEP/IAP  and  TEMP.  Elements  of  the 
developmental  evaluation  may  include  but  are  not 
limited  to  the  following: 

.-w  Software  performance  —  determine  how 

well  the  software  supports  system  performance. 

Examples  of  performance  issues  are: 

-  System  response  time  is  evaluated  for 
conformance  to  specified  time  tolerances. 
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-  System  accuracy  is  evaluated  by  correctness 
of  system  level  decisions  and  the  proximity 
of  computations  to  expected  results. 

-  Recovery/restart  procedures  are  evaluated 
to  ensure  that  users  can  overcome  potential 

.  processing  malfunctions. 

-  Conversion  processes  are  evaluated  to 
ensure  that  data  handling  procedures  are 
described  and  executed  in  a  correct  manner 
(e.g.,  loft  of  baseline).  A  similar 
evaluation  must  also  be  performed  for 
processing  which  generates  the  converted  or 
initial  database  into  the  new  system  format 
(e.g.,  right  of  baseline).  File  contents 
resulting  from  the  conversion  process  must 
be  verified  and  validated  according  to  the 
methodology  and  criteria  specified  in  the 
DT  test  plan. 

(b)  Interoperability  —  the  degree  to  which 
data  is  correctly  exchanged  and  interpreted  between 
systems.  It  is  very  important  to  test  interoperability 
using  actual  target  systems.  The  Army  Interoperability 
Networ)c  (AIN)  at  Ft.  Monmouth,  NJ,  is  available  for 
performing  interoperability  testing  between  actual 
target  systems.  Basically,  there  are  two  types  of 
interoperability:  intrasystem  and  intersystem. 

Examples  of  interoperability  issues  are: 

-  Evaluate  the  acceptance  of  legal 
transmissions  and  rejection  of  illegal 
transmissions. 

-  Evaluate  whether  prioritization  of 
transmissions  is  done  correctly. 

-  Evaluate  how  the  system  performs  under 
load. 

-  Evaluate  how  well  the  system  user  is  able 
to  manage  the  system  to  include  interactive 
terminal  interface,  cycle/system  set-up, 
and  input /output  control. 

-  Evaluate  the  interface  considerations  with 
respect  to  ease  of  data  handling  through 
cycle  processing,  inter-system  data 
transfer,  transmission  of  data  over 
communications  lin)cs,  and  time  sharing 
linJcs  are  functioning  properly. 

(c)  Usability  —  The  effort  required  to  learn 
the  human  interface  with  the  software,  to  prepare  input 
and  to  interpret  output  of  the  software.  Usability 
evaluation  addresses  the  man-machine  interface. 

Examples  of  usability  issues  are: 

-  Evaluate  system  response  to  user 
interaction.  Systems  should  accept  legal 
entries  and  reject  illegal  entries  without 
any  system  degradation. 
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-  Evaluate  output  products  such  as  terminal 
displays,  hard  copy  reports,  magnetic  tape 
and  direct  access  files  are  correct  and 
disposition  for  these  products  are  clear 
and  adequate. 

Cd)  Maintainability  ~  The  effort  required  to 
modify  an  error  in  the  software.  Maintainability  is 
evaluated  by  assessing  the  quality  of  software 
documentation  and  quality  of  code.  The  results  of 
maintainability  evaluation  conducted  by  the  PDSS 
personnel  should  be  used  here.  Examples  of 
maintainability  issues  and  how  they  are  evaluated  are: 

-  Documentation  quality  is  determined  by  its 
completeness,  correctness,  and  traceability 
(i.e.,  determine  if  the  requirements  are 
implemented  in  the  design  and  if  the  design 
is  implemented  in  the  code) . 

-  Code  quality  is  measured  by  programming 
style  (e.g.,  complexity,  modularity  and 
commenting) ,  reserve  memory  capacity  and 
software  metrics.  The  ability  of  the 
developer  to  identify  and  correct  software 
problems  indicates  how  maintainable  the 
software  is. 

-  Reserve  memory  and  processor  capacities  are 
evaluated  to  ensure  they  are  adequate  to 

for  anticipated  expandability. 

”  Training  is  evaluated  to  ensure  the 

adequacy  of  appropriate  training  manuals, 
classroom  and/or  on-the-job  instruction, 
and  problem  reporting  procedures,  if 
applicable. 

(3)  The  developmental  test  report  is  written  by 
the  independent  developmental  tester  and  submitted  to 
the  independent  developmental  evaluator  and  PM. 


8“4.  Metrics  during  developmental  test  and  evaluation 
The  metrics  with  a  check  mark  apply  at  this  time: 


/  Cost 
/  Schedule 
/  CRU 
/  SEE 


/  Reqts  Traceability 
/  Reqts  Stability 
/  Design  Stability 
/  Complexity 


/  Breadth  of  Testing 
/  Depth  of  Testing 
/  Fault  Profiles 
/  Reliability 


a.  Data  from  any  of  the  metrics  may  be  used  to 
assist  in  determining  readiness  for  developmental  test. 

b.  Of  particular  interest  are  breadth  and  depth  of 
testing,  reliability  and  fault  profiles  as  they  pertain 
to  the  activities  performed  during  the  developmental 
test.  The  traceability  and  stability  metrics  and  CRU 
contribute  towards  the  maintainability  evaluation. 
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8-5.  Daeiaioa  eritaria 

a.  All  systems  use  an  independent  evaluation 
report/independent  assessment  report  (lER/IAR)  and  a 
test  report.  The  lER/IAR  is  the  evaluation  of  how  well 
the  system  met  the  critical  issues  stated  in  the  lEP. 
This  evaluation  should  include  software  metrics  and  any 
other  outstanding  software  problems.  The  items  needed 
to  exit  DT  and  enter  operational  testing  are  a 
developmental  test  report  and  successful  completion  of 
the  operational  test  readiness  review  (OTRR) .  To  reduce 
risk,  lERs/IARs  should  be  complete  prior  to  an  lOT  and 
preferably  prior  to  any  OT.  A  favorable  test  result  is 
a  necessary  condition  to  make  the  decision  to  enter  OT. 

b.  If  pT  is  to  be  followed  by  OT,  there  may  be  no 
open  priority  1  or  2  software  problem  reports,  lAW  DOD- 
STD-2167A,  Appendix  C.  Software  problems  fixed  during 
DT  require  evidence  of  comprehensive  regression  testing 
prior  to  admittance  to  operational  testing.  This 
includes  updates  to  documentation  and  impacts  of  any 
work-arounds  required.  The  user  may  require  that 
additional  software  problems  be  fixed  prior  to  OT. 
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Cbaptar  9 

oparational  Tasting  (OT) 

Saetion  I 

Introduction  to  Softvara  T&E  During  Oparational  Tasting 
9-1.  Purposa 

a.  This  chapter  defines  methodology  and  procedures 
for  operational  evaluation  and  test  implementation  of 
systems  which  contain  software.  Operational  testing 
focuses  on  how  the  software  supports  the  system 
mission,  and  the  capability  of  the  user  to  employ  the 
system.  Evaluation  of  software  at  this  time  aims  for 
successful  operational  testing  in  the  operational 
configurations,  loads,  and  environments;  to  assist  in 
deteraining  adequacy  of  test  coverage  so  evaluators  are 
confident  in  evaluations  of  operational  effectiveness; 
and  to  use  evaluations  of  the  Government's  ability  to 
provide  adequate  post-deployment  software  support.  ' 

b.  Operational  testing  is  distinct  from  other  tests 
in  these  ways: 

(1)  It  is  conducted  on  systems  with  typical 
users  (e.g.,  troops,  user  organizations)  in  an 
operational  environment  that  is  as  realistic  as 
practical. 

(2)  OT  uses  personnel  with  the  same  sJcills  and 
training  as  those  who  will  operate,  maintain  and 
support  the  system  when  deployed. 

c.  Realistic  operational  environment  includes 
operations  conducted  lAW  the  system's  wartime, 
mobilization,  or  similar  operational  mission  profiles 
which  specify  the  number,  type  and  frequency  of  combat 
operations  or  specific  transactions/processing  during  a 
period  of  time.  The  test  design  used  in  OT  should  use 
the  SOPs,  tactics  and  doctrine  (if  applicable), 
logistics  and  maintenance  support  concepts  planned  for 
use  when  the  system  is  deployed/ fielded.  The  security 
level  and  threat  levels,  as  defined  in  the  ORD  or  MNS^ 
represent  the  capabilities  postulated  in  the  field. 

d.  OT  can  provide  data  not  obtainable  through  other 
sources  and  may  be  used  to  validate  previous  analyses 
(e.g.,  OAs  based  on  informal  OTs  or  DT) . 

9-2.  Objective 

The  objective  of  operational  testing  and  evaluation  is 
to  determine  the  extent  to  which  the  software  supports, 
and  can  continue  to  support,  the  operational 
requirements  of  the  user.  A  large  portion  of  the 
functions  performed  by  software  in  automated  systems 
(battlefield  or  information)  cannot  be  directly 
observed  by  a  user  or  system  operator.  The  sheer 
amount  and  complexity  of  software  in  these  systems 
precludes  exhaustive  testing  during  operational 
testing.  The  objective  of  this  methodology  is  to 
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ensure,  to  the  naximuin  extent  feasible,  that  user- 
essential  system  functions  are  not  compromised  by 
faulty  software  products  and  that  functions  performed 
by  software  are  done  properly  and  consistently. 

9-3.  Operational  testing 

a.  Operational  testing  varies  with  the  time  period 
during  which  the  tests  occur.  Early  in  the  overall 
software  and  system  development,  the  user  and  the 
developer  may  elect  to  plan  and  execute  tests  and 
evaluations  which  focus  on  specific  issues  of  concern. 
These  operational  tests  are  called  FDTE,  EUTE,  or 
limited  user  tests  (LUT) .  For  example,  new  systems 
which  automate  formerly  manual  functions  represent 
considerable  challenge  to  the  using  community. 

Inforaal  operational  or  limited  user  tests  are  planned 
specifically  to  meet  this  challenge.  The  objectives  of 
these  tests  and  evaluations  are: 

(1)  Exploring  the  tactics  and  doctrine,  as 
<S®fined  in  the  ORD,  for  using  the  system. 

(2)  Determining  how  best  to  implement  the  user 
interface. 

(3)  Determining  how  to  divide  the  workloads 
between  users  at  different  sites  or  using  different 
portions  of  the  system. 

(4)  Assessing  the  standard  operating  procedures 
for  using  the  system. 

(5)  Getting  user  feedback  on  understanding  the 
system  (e.g.,  data  inputs,  error  messages,  screen 
designs,  how  the  work  flows,  the  transactions,  how 
other  systems  provide  inputs,  how  to  handle  recovery, 
what  are  the  minimum  essential  functions  to  perform  the 
lob) . 

(6)  Exposing  the  user  to  the  system  as  it 
develops  to  ensure  user  feedback  and  system  designs 
which  meet  needs  and  expectations. 

(7)  In  the  accelerated  development/ fielding 
acquisition  strategy  a  limited  user  test  is  conducted 
to  prove  out  the  operational  testbed.  This  testbed  is 
comprised  of  the  target  hardware  and  NDI  software  (such 
as  commercial  operating  systems,  database  management 
systems,  etc.)  that  form  the  basis  of  the  operational 
system.  No  application  software  is  tested  at  this 
time.  As  each  application  software  block  is  developed 
It  undergoes  an  initial  operational  test  on  the 
testbed. 

b.  Results  of  informal  operational  or  limited  user 
tests  are  assessed  and  evaluated  to  a  specific  set  of 
issues  and  criteria  for  input  to  decisions  (Milestones 
I  and  II)  prJLbX  to  the  production/deployment  milestone. 
Evaluations  of  these  tests  may  be  in  the  form  of  an  OA 
or  an  AOA  prepared  by  the  independent  operational 
evaluator.  Results  are  used  to  feed  back  into  the 
system  design  and  implementation. 
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c.  Informal  operational  or  limited  user  tests 
cannot  replace  the  initial  operational  test  (lOT)  used 
for  the  Milestone  III  decision.  However,  they  are  used 
to  provide  supplemental  data,  thereby  reducing  the 
scope  of  the  formal  operational  tests.  Formal 
operational  tests  and  their  associated  evaluations,  if 
required  lAW  AR  73-1  and  AR  25-3,  are  crucial  elements 
of  the  review  process  which  results  in 
production/deployment  decisions  at  Milestone  III. 

Formal  operational  tests  have  been  called  by  terms  such 
as  operational  test  (OT) ,  initial  operational  test 
(lOT),  follow-on  operational  test  (FOT) ,  multi-service 
operational  test  (MOT) ,  joint  test  (JT) ,  system 
acceptance  test  or  software  acceptance  test  (SAT). 
Formal  OT  is  required  and  conducted  on  all  Aray  systems 
containing  software,  including  executive  software,  new 
developments,  NDI  and/or  those  in  the  PDSS  phase.  OT, 
under  specific  conditions  outlined  in  AR  73-1  and  Part 
Five  of  DA  Pamphlet  73-1,  may  be  combined  with 
developmental  testing.  The  extent  and  focus  of  the 
formal  OT  is  determined  by  the  life  cycle  phase  of  the 
system,  whether  independent  evaluation  is  required,  the 
acquisition  category  of  the  system,  and  whether  the 
system  is  on  the  DOD  Oversight  List. 

9-4.  Entry  objectives 

a.  Operational  test  readiness  statement  (OTRS) . 

The  PM,  user  representative,  and  trainer  provide 
statements  that,  in  their  area  of  responsibility,  the 
system  is  ready  to  undergo  OT.  The  topics  discussed 
are  in  the  OTRS  part  of  the  information  presented 
during  OTRRs. 

b.  Operational  test  readiness  software  assessment. 
The  evaluator  assesses  the  ability  of  the  software  to 
support  the  objectives  of  the  operational  test  before 
committing  resources  for  its  execution.  This  OT 
readiness  assessment  involves  the  evaluation  of  whether 
the  software  to  be  used  in  the  OT  will  run  under  the 
scenarios  and  types  of  conditions  planned. 

Developmental  testing,  software  requirements 
verification  testing  and  other  forms  of  software 
testing  (Government-witnessed  IV&V,  for  example)  are 
used  to  support  the  assessment  of  the  software 
readiness  for  OT.  However,  before  such  tests  can  be 
considered,  the  priority  software  requirements  and 
conditions  for  test  are  compared  to  the  subject  test 
plans.  This  comparison  will  be  rigorous  enough  to 
provide  confidence  that  the  software  will  not  degrade 
or  terminate  system  performance  in  the  upcoming  OT. 

(1)  Completed  software  developmental  tests.  The 
operational  evaluator  ensures  that  the  Government 
developmental  testing  conducted  is  sufficient  to 
support  the  OT  readiness  assessment.  The  final  review 
of  the  developer's  technical  and  Government 
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developmental  test  results,  as  presented  by  the  PM, 

IV&V  agent,  and  the  users,  and  the  acceptance  of  the 
software  for  use  in  OT  is  done  at  the  OTRR. 

(2)  OT  version  description.  The  operational 
evaluator  includes,  as  part  of  the  OT  readiness 
assessment,  a  clear  statement  of  the  version  of  the 
software  to  be  used  in  operational  test.  Any 
deviations  in  that  version  from  the  design  software  or 
from  the  software  intended  for  fielding  and  the  impact 
of  any  omissions  or  deviations  in  the  test  or  the 
evaluation  will  be  described.  Systems  which  have 
excessively  patched  software,  incomplete  software 
(e.g.,  lacking  rec^ired  functions),  software  which  is 
not  lAW  its  associated  documentation,  or  software 
lacking  tight  change  control  will  not  be  allowed  to 
proceed  to  OT.  The  software  configuration  management 
(CM)  that  is  used  by  the  software  developer  Should 
identify  the  anticipated  schedule  for  software  releases 
and  publication  of  software  documentation.  In 
preparation  for  the  OTRR,  the  organization  responsible 
for  CM  is  tasked  to  provide  the  operational  evalu-* 
with  the  OT  software  vers-n  description.  The 
operational  evaluator  suKj>arizes  this  description  in 
the  OTRR  briefing. 

c.  Software  problem  reports.  Priority  l  and  2 
software  problem  reports  must  be  resolved  prior  to  OT. 
The  system  may  be  allowed  to  proceed  to  OT  with  open 
problem  reports  (priorities  other  than  1  or  2)  provided 
concurrence  is  obtained  from  the  user  representative. 
Work-arounds  may  be  necessary  for  these  open  problem 
reports  in  order  to  avoid  delaying  the  start  of  OT. 
However,  these  must  be  clearly  specified,  trained  .or, 
and  accepted  by  the  user  representative. 

d.  Operational  test  readiness  review  (OTRR). 

(1)  OTRRs  are  required  for  formal  operational 
tests  whether  independently  evaluated  or  not.  Informal 
operational  or  limited  user  tests  may  also  require 
OTRRs  prior  to  test  conduct  as  detailed  in  the  TEMP. 

The  results  of  the  operational  test  readiness 
assessment  are  presented  and  metrics  repozrted.  A 
summary  of  the  planned  OT  test  scenarios  and  status  of 
instrumentation,  if  any,  are  presented.  The  relevance 
of  all  unresolved  critical  and  serious  software 
trouble/problem  reports  is  discussed  in  terms  of  their 
potential  impact  on  the  planned  scenarios  or  required 
user  work-arounds.  The  PM^  user  representative, 
operational  tester,  and  operational  evaluator,  at  a 
minimum,  participate  in  the  OTRR.  Other  organizations 
participate  at  the  invitation  of  the  evaluator  and 

ter ter  organizations.  The  security  accreditation 
agent,  IV&V  agent,  and  developmental  evaluator 
participate  if  needed. 

(2)  Part  Five  of  DA  Pamphlet  73-1  provides  the 
OTRR  procedure  and  sample  agenda.  The  acquisition 
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category ,  system  complexity  and  interest  level 
determine  the  extent  and  formality  of  OTRRs.  There  are 
several  OT^s  which  precede  the  final  OTRR,  each  with 
similar  objectives:  to  provide  an  iterative  assessment 
of  software  and  system  readiness  for  testing,  and  to 

provide  an  update  on  the  readiness  of  the  operational 
test  planning. 


9-5.  Doeumantation 

a.  The  evaluation  plans,  test  plans  and  supporting 
plans  must  be  approved  and  ready  to  support  the 
execution  of  operational  testing  and  evaluation.  The 
level  of  approval  for  the  plans  depends  upon  the  system 
acquisition  category,  approval  authority  and  visibility 
(e.g. ,  systems  on  the  DOD  oversight  list  are  reviewed 
and  plans  for  an  lOT  must  be  approved  by  DOD) . 

b.  For  full  evaluation  and  abbreviated  evaluation 
systems,  the  following  plans  are  used  and  are  approved 
for  execution  during  this  T&E  event.  The  document 
prepared  by  the  operational  evaluator  is:  the  test  and 
evaluation  plan  (Chapters  l  and  2  of  the  TEP) .  The 
documents  prepared  by  the  operational  test’ organization 
are:  the  test  plan  (Chapter  3  of  the  TEP)  and  the 
detailed  test  planning  dociiment  (DTP  or  test  plan  lAW 


c.  Formal  (OT)  including  test  planning  and  test 
reporting  are  still  performed  for  systems  that  are  not 
required  to  conduct  independent  evaluations. 
Documentation  is  less  extensive,  but  the  requirement  to 
perform  adequate  testing  remains  constant.  Major 
command  (MACOM)  level  approval  systems  or  installation 
approval  systems  must  conduct  operational  testing  to  an 
established  set  of  user  requirements,  in  the 
operational  environment,  just  as  do  the  higher  approval 
level  systems.  Documentation  prepared  by  the 

tester  consists,  at  a  minimum,  of  the  TEP, 
DTP  or  test  plan  lAW  DOD-STD-7935A,  and  the  test 
analysis  report  or  test  report. 

d.  The  level  of  documentation  for  the  operational 
tests  varies  over  the  life  cycle.  Formal  operational 
tests  will  use  final,  validated  documentation:  user's 
manuals,  end  user's  manuals  and/or  computer  operation 
manuals,  training  documents,  technical  manuals,  and 
continuity  of  operations  plans  (COOP  or  CONOPS) . 

Systems  which  are  independently  evaluated  will  assess 
the  maintainability  evaluation  results  conducted  by  the 
PDSS  personnel.  Documentation  such  as  maintenance 
manuals,  software  programmer's  manuals  (SPM) ,  firmware 
support  manuals  or  similar,  requirements  and  software 
specifications  (e.g.,  FD,  US,  SRS,  IRS,  SS, 

system/ segment  design  document  (SSDD) ,  version 
description  document  (VDD) ,  SPS,  etc.)  must  be  ready 
for  PDSS.  -Transition  plans  must  be  completed,  the 
supportability  statement  must  be  approved,  and 
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configuration  inanageaent  plans  must  be  in  place  and 
working. 

9-6.  Activities 

a.  Test. 

(1) _  lOT  or  FOT  is  a  system-level  test  performed 
by  a  test  activity  independent  of  the  developer,  the  PM 
(program  sponsor)  or  the  user  proponent.  Other  OTs  are 
usually  conducted  to  support  developments  for  the 
users'  representative  or  PM  and  do  not  require  the 
rigorous  independence  of  an  lOT  or  FOT.  When  the 
accelerated  development/ fielding  acquisition  strategy 
is  used,  the  full  level  of  detail  described  in  this 
section  applies  to  the  fielding  certification  lOT  of 
the  representative  sample,  lOT.C.  Not  all  aspects  of 
this  section  apply  equally  to  the  incremental  software 
block  lOTs  that  precede  or  follow  the  lOT.C  as  they  are 
dependent  on  the  functionality  of  each  block. 

(2)  Characteristically,  the  test  is  executed  in 
a  realistic  or  live  environment  representative  of  the 
operational  system  mission,  by  the  designated  system 
users.  The  OT  is  conducted  on  the  specified  target 

ha  dware,  in  the  production-representative  system 
cc.  1  iguration  (e.g.,  communications,  peripherals,  other 
system  interfaces,  etc.)  with  the  software 
release/version  to  be  fielded. 

(3)  The  objective  of  the  OT  is  not  to  exercise 
and  observe  every  technical  aspect  of  the  system,  but 
rather  the  entirety  of  the  system  and  its  planned 
support  environment  when  it  is  deployed.  The  structure 
of  the  test  considers  elements  of  effectiveness  and 
suitability  (e.g.,  training,  logistics,  human  factors): 

(a)  Training  effectiveness,  especially  in  the 
area  of  embedded  software  which  will  be  used  to  train 
the  users; 

(b)  Usability/understandability  of  the 
software  interfaces  by  the  user; 

(c)  Capability  of  the  software  to  support  the 
system  mission; 

(d)  Capability  of  the  user  to  recover  and 
restore  the  system  when  malfunctions  occur  (e.g., 
maintainability) ; 

(e)  System  and  software  documentation; 

(f)  User  performance  requirements 
(transaction  processing  time,  peak  processing  loads  and 
conditions,  transition  to  mobilization  processing  from 
peacetime,  combat  modes,  etc.); 

(g)  Capability  of  the  system  to  perform 
within  the  accuracy  required  by  the  users  as  dociimented 
in  user  requirements  (e.g.,  MNS,  FD,  ORD) ; 

(h)  Capability  to  support  the  operational 
mission  over  the  time  periods  required  (e.g., 
robustness,  fault  tolerance,  reliability) ; 
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(i)  Capability  to  operate  in  degraded  inodes 
(users  specify  the  minimum  functions  which  must  be 
available) ; 

(j)  Capability  to  provide  CONOPS  or  COOP; 

()c)  Capability  of  the  system  to  interface  and 
interoperate  with  other  distinct  systems  as  required. 

(4)  Software  change  policy  during  OT.  Software 
found  to  be  deficient  to  the  extent  that  objectives  of 
the  OT  cannot  be  met,  will  require  a  special  OTRR  to 
review  the  extent  of  modifications  required,  the  impact 
of  modifications  on  the  ability  of  the  test  to  meet  its 
objectives,  and  the  resource  implications  associated  if 
the  completed  portion  of  the  test  has  to  be  repeated. 

A  software  modification  is  allowed  only  if  directed  by 
the  commander  of  the  test  proponent  agency  in 
coordination  with  the  evaluation  agency. 

(5)  Operational  testing  shall  occur  in  parallel 
with  the  existing  systems.  If  this  is  not  feasible,  a 
written  waiver  will  be  prepared  by  the  PM  and 
operational  tester,  and  submitted  to  the  system 
approval  authority  (e.g.,  MAISRC,  DA).  The 
system/software  will  normally  be  removed  from  the  test 
site  at  the  close  of  operational  testing.  A  Milestone 
III  production/deployment  decision  and  a  materiel 
release  approval  lAW  AR  700-142  constitute  the 
authority  to  field  the  new  system. 

(6)  Metrics  collected  and  analyzed  for  OT  are 
reliability  and  breadth  of  testing.  Fault  discovery 
and  closure  data  are  analyzed  using  the  priorities  in 
DOD-STp-2167A,  Appendix  C.  Further  discussion  of  the 
reliability  and  breadth  of  testing  metrics  is  in 
Chapter  17,  Software  TiE  Metrics,  of  this  part  of  DA 
Pamphlet  73-1. 

b.  Evaluation.  The  primary  objective  of 
operational  evaluation  is  to  address  the  operational 
effectiveness  and  suitability  of  Army  systems  for  use 
by  typical  users  in  realistic  operational  environments. 
The  scope  of  operational  evaluation  does  not  extend  to 
the  technical  aspects  of  the  software,  but  rather  to 
the  ability  of  the  software  to  support  the  operational 
mission  of  the  system  as  a  whole.  Although  the 
evaluation  focuses  on  the  critical  operational  issues 
and  criteria  identified  in  the  TEMP,  it  is  not  limited 
to  them.  It  provides  estimates  of  the  expected  impacts 
upon  system  performance  in  terms  of  the  ability  to  meet 
the  mission  need.  Even  in. those  situations  when  DT  and 
OT  are  conducted  concurrently,  separate  evaluations  (TE 
&  OE)  will  be  performed  by  separate  evaluation 
organizations.  As  a  minimum,  the  following  factors 
will  be  addressed  in  the  evaluation: 

(1)  Capability  of  the  software  to  support  the 
system  mission. 

(2)  System  user’s  ability  to  manage  the  system 
(cycle/system  set-up,  and  input/output  control) . 
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(3)  Adequacy  of  interfaces  with  respect  to  loss 
of  data  handling  through  processing,  inter-system  data 
transfer,  and  two-way  transmission  over  communications 
links. 

(4)  Adequacy  of  control  statements  to  reflect 
the  required  functional  order  of  cyclic  and  system 
processes. 

(5)  That  system  products  such  as  reports  and 
magnetic  media  have  been  produced,  are  correct,  and 
disposition  and  handling  instructions  for  these  are 
clear  and  accurate. 

(6)  That  system  performance  meets  user-defined 
performance  requirements  as  stated  in  the  ORD  and  the 
UFD. 

(7)  Burden  placed  upon  the  system  or  operator  by 
operator  intervention  requirements. 

(8)  Capability  of  the  system  to  perform  within 
the  accuracy  required  by  the  user  as  documented*  in  user 
requirements. 

(9)  Capability  of  the  system  to  operate  for  the 
time  periods  required  (e.g.,  robustness,  fault 
tolerance,  reliability) . 

(10)  Capability  to  operate  in  degraded  mode 
(user  specified  minimum  functions) . 

(11)  That  recovery /restart  procedures  overcome 
processing  malfunctions. 

(12)  The  level  of  documentation  is  appropriate 
such  that  user  personnel  can  comprehend  and  use  it 
effectively. 

(13)  That  file  conversion  procedures  are 
adequately  described  and  executed. 

(14)  That  training  and  training  materials  are 
adequate  to  produce  capable,  competent  system 
users/operators. 

(15)  Capability  to  provide  CONORS  or  COOP. 

9-7.  Decision  criteria 

a.  In  order  for  the  system  to  successfully  complete 
this  phase,  the  following  documentation  must  be 
complete:  an  approved  test  and  evaluation  report 
(TER).  The  TER  is  based  on  the  test  results,  and  any 
other  documentation  supporting  the  independent 
operational  evaluator's  (lOE)  evaluation  of  the  system 
effectiveness  and  suitability.  It  identifies  how  well 
the  system  met  the  COIC.  If  the  system  is  found  to  be 
effective  and  suitable,  the  T&E  findings  support  a 
production  decision  or  software  distribution. 

b.  Evidence  of  failure  to  meet  COIC  or  other 
problems  which  preclude  the  accomplishment  of  the 
intended  missions  will  be  the  basis  for  a  decisi:Qn  by 
the  appropriate  decision  authority  and  review  councils 
(i.e.,  ASARC  or  MAI SRC ) .  Among  the  possible 
alternative  courses  of  action  are: 

(1)  To  implement  follow-on  formal  operational 
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test(s).  The  emphasis  behind  these  follow-on  tests 
will  be  to  address  the  problems  identified  during 
initial  operational  testing. 

(2)  To  return  the  program  to  the  DT  phase  for 
significant  corrective  action. 

(3) -  To  cancel  the  program. 

c.  If  follow-on  testing  is  required,  the  process 
for  development  of  test  and  evaluation  plans  previously 
discussed  must  be  re-initiated  to  address  only  those 
problem  areas  in  the  TER.  At  the  completion  of  follow- 
on  formal  operational  test,  the  decision  authority  will 
make  a  production  decision  based  upon  all  information 
Available  to  that  body  at  the  time,  to  include  the 
recommendation  of  the  independent  operational  evaluator 
if  solicited. 

d.  Post-deployment  software  support  tests  are 
discussed  in  Chapters  11-16,  post  deployment  support 
test  and  evaluation  procedures,  of  this  part  of  DA 
Pamphlet  73-1. 
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Chaptar  10 

Matarial  Ralaasa  for  Softvara 
10-1.  Introduction 

The  materiel  release  process  is  intended  to  assure  that 
Army  materiel  is  suitable  and  supportable  before  it  is 
issued  to  the  users.  The  materiel  fielding  and 
transfer  processes  ensure  the  orderly  and  effective 
deployment  and  transfer  of  Army  equipment,  including 
all  necessary  logistic  support  requirements.  AR  700- 
142  documents  the  Army's  materiel  release,  fielding, 
and  transfer  processes.  These  processes  pertain  to 
systems  as  a  whole,  which  can  be  comprised  of  either 
hardware,  software,  or  both.  AR  700-142  addresses  and 
emphasizes  the  need  to  fully  assess  release/fielding 
issues  relating  to  both  constituents,  and  further 
identifies  and  emphasizes  the  software  aspects.  This 
chapter  expands  -upon  these  software  issues  as  they 
pertain  to  the  release  and  fielding  process. 

10-2.  Objectives 

The  objectives  of  the  materiel  release  for  issue 
process  are  to: 

a.  Establish  a  management  control  system  to  ensure 
that  materiel  released  for  issue  by  the  Army  is  safe, 
operates  as  designated,  and  is  logistically 
supportable. 

b.  Provide  a  system  which  enables  Headquarters, 
Department  of  the  Army  (HQDA)  to  have  overall 
visibility  and  control  of  the  materiel  release  process. 

c.  Provide  a  mechanism  to  monitor,  control  and 
follow  through  on  all  conditional  releases  until  full 
release  is  obtained. 

10-3 .  8cope 

a.  Materiel  subject  to  release  actions  under  AR 
700-142  include: 

(1)  First  time  procurements,  including  depot 
assembly  programs,  developmental,  non-developmental  and 
product- improved  systems,  and  non-major  systems 
governed  by  AR  25-1,  AR  25-5,  AR  40-60,  AR  70-1,  AR  70- 
15,.  AR  105-7,  and  AR  350-38  for  which  the  Army  has  life 
cycle  materiel  management  responsibility.  Software 
that  is  part  of  a  new  system  or  is  part  of  a  hardware 
and/or  firmware  change  is  released  as  part  of  the  prime 
end  item. 

(2)  Follow-on  procurements  of  systems  which  have 
been  previously  issued  under  full  release  (e.g., 
without  a  break  in  production  of  one  year  or  more) , 
follow-on  procurements  produced  by  a  different 
contractor,  or  systems  currently  under  conditional  or 
training  release. 
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(3)  Conversion  programs  for  which  a  change  in 
type  classification  model /type  occurs,  or  configuration 
changes  significantly  affect  form,  fit,  or  function. 

(4)  Changed  software,  to  include  embedded, 
proprietary,  and  NDI  software  that  meets  one  or  more  of 
the  following  criteria; 

(a)  Software  that  significantly  changes  (or 
has  the  potential  to  change  if  not  adequately  tested) 
mission  function,  capability,  performance  parameters, 
interoperability  requirements,  reliability  or  safety. 

(b)  A  bloc}c  update  consisting  of  a  software 
change  of  more  than  30%  executable  lines  of  code  (LOC) 
or  30%  executable  cumulative  LOC  changes  not  having 
required  release  approval  since  the  last  materiel 
release.  These  criteria  may  be  tightened  at  the 
discretion  of  the  HATDEV  based  upon  the  criticality  of 
the  software  changes. 

(c)  A  bloc]c  update  consisting  of  a  software 
translation  of  30%  LOC  to  a  different  computer 
programming  language . 

(d)  Deployable  system  integration  software, 
which  is  newly  developed  software  added  to  NDI  software 
necessary  to  billow  customization  and  interoperability, 
or  simply  to  integrate  multiple  NDI  software  items  so 
that  they  work  together  under  one  "system." 

(e)  Software  that  is  significantly  changed  to 
run  on  a  different  computer  processor. 

(f)  Software  changes  that  require  new  user- 
level  test  equipment  and/or  impact  25%  of  the  program 
of  instruction. 

(5)  All  associated  support  items  of  equipment 
(ASIDE)  and  ancillary  equipment  comprising  the  total 
materiel  system. 

(6)  Stand-alone  or  embedded  automatic  data 
processing  equipment  (hardware  and  software)  including 
automated  test  equipment  (ATE) ,  deployed  to  the  user 
MACOM. 

(7)  Materiel  manufactured  by  a  second  source. 

(8)  Equipment  subject  to  mission  support  plan 
(MSP)  for  which  lERs/IARs  are  required. 

b.  The  process  as  discussed  herein  applies  to  the 
materiel  discussed  in  paragraph  10-3 a  above,  with  the 
following  special  provisions  noted: 

(1)  Materiel  developed  by  the  Army,  procured  by 
the  Defense  Logistics  Agency  (DLA) ,  and  distributed  by 
the  Army  requires  a  materiel  release  action. 

(2)  For  materiel  developed  by  the  Army  and 
assigned  to  the  DLA  or  the  General  Services 
Administration  (GSA)  for  procurement  and  distribution 
(integrated  materiel  management),  the  Army  MATDEV  will 
establish  a  memorandum  of  understanding/agreement.  All 
necessary  technical  and  logistical  support  will  be 
provided  to  DLA  or  GSA  to  assure  distrib  'tion  of  the 
materiel. 
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(3)  For  materiel  developed  by  the  Army  for 
another  service,  federal  agency,  or  foreign  government, 
the  criteria  specified  in  the  agreements  between  the 
Army  and  the  user  or  developer  will  govern  the  materiel 
release  process.  To  the  extent  that  those  criteria  are 
not  defined,  the  criteria  in  AR  700-142  will  apply. 

(4)  Security  assistance  programs  may  be  waived 
from  the  instruction  specified  herein  to  accommodate 
the  terms  of  an  agreement  or  when  reguested  by  the 
customer . 

10-4.  Materiel  release  policy 

a.  Materiel  approved  for  release  must  be  safe, 
operationally  effective,  operationally  suitable,  and 
logistically  supportable. 

b.  The  lead  MATDEV  responsible  for  fielding  the 
prime  end  item  of  a  materiel  release  system  is  also 
responsible  for  ensuring  the  availability  of  the 
support  equipment,  to  include  materiel  system  computer 
resources,  initial  support  resource,  ammunition,  and 
ASIDE. 

c.  Materiel  proposed  for  release  will  remain  under 
the  control  and  accountability  of  the  materiel 
developer  until  the  release  approval  is  granted. 
Materiel  may  be  pre-positioned  before  materiel  release 
is  approved,  but  the  final  transfer  of  accountability 
and  control  to  the  user  will  occur  only  after  release 
approval  is  obtained.  Materiel  release  requests  should 
be  initiated  approximately  180  days  prior  to  first  unit 
equipped  (FUE)  or  hand-off  date  to  allow  materiel 
release  approval.  The  materiel  release  decision  should 
occur  by  120  days  prior  to  first  unit  equipped  delivery 
(FUED)  to  prevent  turbulence  in  the  materiel  fielding 
process. 

d.  The  type  of  release  (full,  conditional  or 
training)  is  determined  after  a  comprehensive 
assessment  of  the  total  materiel  system  (see  DA 
Pamphlet  700-142) .  If  any  ASIDE  is  conditionally 
released  to  the  lead  MATDEV,  the  materiel  system 
release  must  also  be  conditional. 

e.  The  existence  of  a  safety  deficiency  in  a  system 
does  not  preclude  full  release  of  the  system  in  cases 
where  the  Army  Acquisition  Executive  (AAE)  or  the  AAE 
designee  has  accepted  the  associated  ris]c.  Acceptance 
of  the  risJc  is  documented  in  the  system  safety  ris)c 
assessment. 

f.  Prior  to  the  release  decision,  the  MATDEV  will 
provide  the  logistician,  user's  representative,  and 
other  participants  in  the  Materiel  Release  Review  Board 
(MRRB) ,  a  copy  of  the  documentation  showing  that  the 
materiel  release  (training,  conditional,  or  full 
release)  pre-requisites  have  been  met. 


10-3 


30  8«pt«Bb«r  1992 


DA  Pam  73-1,  Part  Savam  (Draft) 


10-5.  Typas  of  matarial  ralaasa 

Materiel  release  falls  into  one  of  three  categories: 
full,  conditional,  or  training.  These  are  discussed 
below: 

a.  Full  release.  A  full  release  is  authorized 
after  the  jnateriel  has  been  tested  and  evaluated,  and 
determined  to  meet  all  established  requirements.  The 
user  MACOM  must  also  concur  with  the  final  materiel 
fielding  plan,  and  all  other  aspects  of  the  logistics 
support  system  must  have  been  achieved  as  specified  in 
the  integrated  logistic  support  plan  (ILSP)  approved  at 
the  Milestone  III  decision  review.  In  addition, 
validated  and  verified  draft  DA  equipment  publication 
(or  authenticated  commercial  manuals) ,  and  all  classes 
of  supply  and  sustainment  materiel  must  be  available 
prior  to  or  concurrent  with  fielding. 

b.  Conditional  release.  A  conditional  release  may 
be  authorized  when  one  or  more  of  the  criteria  for  full 
release  have  not  been  met.  If  an  urgent  need  exists 
for  the  materiel,  a  conditional  release  may  be  granted 
as  long  as  the  hardware  and/or  software  is  available 
and  acceptable  to  the  user  MACOM  (requires  general 
officer  acceptance),  and  if  an  interim  means  of  support 
exists.  However,  a  conditional  release  requires  that  a 
get -well  plan  (for  each  condition  that  precluded  a  full 
release)  be  developed  and  approved,  and  that  the 
conditional  release  be  restricted  to  a  specific 
quantity,  location,  and  application.  Requests  for 
conditional  release  of  DOD  major  and  defense 
acquisition  program  (DAP)  materiel  systems  will  be  sent 
through  HQDA  (DALO-SMS)  to  the  Vice  Chief  of  Staff, 

Army  (VCSA) .  Status  must  be  reported  periodically 
until  all  conditions  are  satisfied  and  full  release  is 
realized. 

c.  Training  release.  Training  release  is  the 
release  of  materiel  to  TRADOC  or  a  TRADOC-sponsored 
school  for  training  use  only.  Prior  to  approving 
training  releases,  the  MATDEV  will  ensure  that  critical 
issues  such  as  safety,  availability  of  spare/repair 
parts,  technical  documentation,  responsibility  for 
maintenance  support,  and  other  conditions  which  limit 
use  of  the  item  are  identified  and  accepted  by  the 
trainer. 

10-6.  Procedures  for  materiel  release 
Full  and  training  releases  are  approved  by  the  MSC  upon 
completion  of  the  materiel  release  process  as  described 
in  the  preceding  paragraphs.  A  copy  of  the  approval 
document  is  provided  to  HQDA.  Requests  for  conditional 
releases  of  ACAT,  and  Director,  Operational  Test  and 
Evaluation  (DOT&E)  oversight  materiel  systems  are 
forwarded  through  HQDA  to  the  VCSA  for  approval,  with 
the  exception  of  AMC-sponsored  systems  which  are 
approved  by  CG,  AMC.  Requests  for  conditional  releases 
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of  the  remaining  IPR  programs  are  approved  by  the 
MATDEV/AMC.  All  requests  for  conditional  release  must 
document  gaining  MACOM  (general  officer)  acceptance  for 
the  conditions  of  release. 

10-7.  Doetunantatioa  for  material  ralaaaa 
To  ensure  that  the  objectives  of  the  materiel  release 
process  are  met,  strict  documentation  must  be  prepared 
and  submitted  for  release  approval.  AR  700-142  details 
the  scope  and  preparation,  but  as  a  minimum  the 
materiel  release  documentation  should  include  an 
independent  evaluation/assessment  from  the 
developmental  and  operational  evaluator/assessor, 
confirmation  of  the  safety  assessment,  statements  of 
supportability  (for  each  hardware /software  major 
constituent) ,  confirmation  that  all  aspects  of  the 
logistics  support  system  plan  agreed  to  at  the 
Milestone  III  decision  have  been  achieved,  and  a  signed 
materiel  fielding  agreement.  Of  significant ‘ note  is 
the  requirement  for  documentation  of  all  software 
supporting  data,  especially  for  re-releases 
necessitated  by  software  modifications/  upgrades.  If 
the  materiel  release  is  conditional,  also  included  is  a 
statement  from  the  MACOM  user  accepting  all  conditions 
for  release,  along  with  general  officer  concurrence 
from  the  gaining  command. 
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Chaptar  ll 

Naads  and  Davalopmant  of  coneapts  in 


11~1*  Introduction 

*  j software  support  (PDSS)  consists 
of  modifications  and  maintenance  of  software  in  fielded 
systems  or  systems  to  be  fielded  after  MS  III  decision. 
Tne  development  of  a  change  follows  the  same  basic  life 
cycle  process  as  a  new  system;  however,  the  scope  and 
resources  involved  may  be  reduced.  PDSS  usually  is  the 
longest  part  of  the  life  cycle.  The  PDSS  environment 
generally  produces  many  small  changes  over  a  period  of 
time  rather  than  a  few  large  changes.  The  PDSS 
organization  must  be  able  to  coordinate  and  phase-in 
these  many  changes  into  a  few  formal  software  releases 
to  minimize  the  formal  processing  reguired  and  avoid 
any  disruption  in  the  fielded  system.  Differences  in 
the  magnitude  and  timing  of  software  changes  must  be 
considered  in  identifying  the  scope  of  T&E  reguired  and 
the  extent  of  T&E  team  involvement.  The  concepts  of 
block  changes  and  block  releases  must  be  integrated 
into  the  T&E  planning,  in  addition,  the  PDSS 
organization  may  be  involved  in  one  of  two  capacities: 
as  the  actual  software  change  developer,  or  as  the  PM 
of  a  contracted  software  change  development,  in  both 
cases,  the  scope  of  the  T&E  reguired  must  be 
selectively  tailored  for  each  situation. 

P^J^POses  of  this  part  of  DA  Pamphlet  73-1, 
PDSS  test  and  evaluation  refers  to  changes  made  to  the 
system  after  first  unit  eguipped  (FUE)  or  first  site 
fielded.  These  changes  are  within  the  authority  of  the 
system  manager  or  MATDEV.  Test  and  evaluation  of 
changes  during  PDSS  and  software  development  activities 
mirror  those  detailed  in  Chapters  5,  6,  7,  8,  and  9. 
c.  PDSS  reguires  some  flexibility  in  development, 
end  evaluation  in  order  to  respond  to  the 
changing  environment.  Often,  new  reguirements  have 
date-driven  implementation  milestones  which  must  be 
accommodated  to  preclude  adverse  impact  on  system 

Emergency  changes  which  must  be  completed 
within  48  hours  may  be  reguired.  The  T&E  team  must  be 
responsive  and  is  an  integral  element  of  effective  PDSS 
support  and  fielding  of  guality  products. 

11-2 .  Reguirements  generation 

a.  Software  changes  to  deployed  systems  are 
generated  because  of  latent  defects,  doctrinal 
reguirements,  threat  changes,  weapon/munitions 
upgrades,  interoperability  reguirements ,  product 
improvements  and  new  system  functions.  Change  reguests 
are  normally  generated  by  the  using  agency,  CBTDEV,  or 
FP  and  forwarded  for  approval,  prioritization  and 
implementation.  Changes  are  generated  and  processed  in 
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^  25-3,  AR  71-9,  AR  70-  7,  AR  70-15 
and  DA  Pamphlet  25-6,  ,  /u  id 

^ergency  changes  frequently  need  to  be  released 
o  the  field  within  48  hours,  in  this  case,  the  T&E 
team  needs  to  respond  quickly  to  ensure  that  the 
software  changes  are  made  and  adequately  retested. 

respond  quickly  to 

(usually  in  the  lab)  and  monitor  the  fix 
once  It  reaches  the  field.  A  good  library  of  test 

adequate  test  instrumentation  is 
very  important.  Because  changing  software  to  fix  a 
problem  can  introduce  additional  problems,  retesting 
(r®gr®ssion)  must  be  done  along  with  testing  the  fix. 

It  IS  permissible  for  formal  dociimentation  updates 
(e.g.,  design  dociiments)  to  catch  up  later.  All 
changes  to  the  software  must  be  identified  and 

configuration  management  function. 
Changes  to  documents  that  are  re(^ired  for  field 
operation  of  the  changed  software  must  be  developed  and 
provided  with  the  changed  software.  Identification  and 
formal  processing  of  all  changes  must  be  accomplished 
as  soon  as  possible,  but  the  formal  publication- ofi-baJne 
support  it®ms  may  be  deferred  until  the  next?.mi3amrffld 
clock  change.  Emergen: y  changes  such  as  this'%ad.l  be 
as  part  of  the  next  planned 
software/system  change  package  or  materiel  change. 

c.  While  all  changes  for  systems  under  development 
must  undergo  validation  and  verification  testing, 

changes  to  deployed  systems  may  not  require 
formal  developmental  testing  or  operational  testing. 
However,  all  emergency  changes  will  undergo  formal 
testing  with  the  next  planned  updates.  The 
system/operations  manager,  with  the  concurrence  of  the 
system  user,  may  only  be  capable  of  performing  limited 
testing  of  emergency  software  corrections  prior  to 
granting  release.  ^ 

Configuration  Control  Board  (CCB) 

requests  are  documented  on 
®^®  ®PPJ^oved  and  scheduled  for 
implementation  by  the  appropriate  configuration -control 
board  lAW  DOD-STD-480B.  Configuration  management, 
release  control,  and  version  updates  are 
further  explained  in  detail  in  Chapters  3,  6,  and  7. 

11-4.  Software  change  paeJeage 

a.  Individual  software  changes,  documented  on  ECP- 
Ss,  are  categorized  based  on  the  urgency  of 
implementation  lAW  DOD-STD-480B  (emergency,  urgent  or 
routine) .  Changes  are  also  classified  as  priority  i 
through  5,  in  accordance  with  DOD-STD-2167A,  Appendix 
^®l®tive  to  the  impact  on  operational  mission 
effectiveness.  The  extent  and  criticality  of  the 
change  to  the  mission  coupled  with  the  urgency  of 
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delivering  the  change  to  using  agencies  may  dictate  the 
extent  and  thoroughness  of  the  test  and  evaluation 
effort. 

11-5.  Raquiramants  documentation  updates 
Requirements  that  drive  changes  to  deployed  software 
originate  from  many  sources:  new  concepts  and 
doctrine,  software  problem  reports,  quality  deficiency 
reports  from  users,  changes  to  interfacing  systems  or 
materiel  changes  in  the  system  hardware.  The  user 
representative  analyzes  these  sources  and  documents 
requirements  for  changes  in  ECP-S.  Ultimately,  the  CCB 
agrees  to  a  packages  of  changes  derived  by  the  MATDEV 
from  ECP-Ss  and  documented  in  Materiel  Change 
Information  Reports  (MCIR) ,  ECPs,  and  other  change 
packages.  Changes  that  affect  only  the  product 
baseline  will  generally  require  changes  to  the  design 
and  test  documentation,  but  will  not  require  changes  to 
the  requirements  documents.  Changes  to  the  functional 
and  allocated  baselines  will  require  changes  to  one  or 
more  requirements  documents,  such  as  the  UFD,  the 
system/segment  specification,  software  specifications, 
interface  specifications,  and  data  base  specifications. 
Test  documents  requiring  update /development  are  the 
test  plan,  test  description  or  conditions,  test 
procedures,  TDP,  and  TEP. 

11-6.  Determining  the  scope  of  T&E  for  changes 

a.  During  development,  all  T&E  functions  are 
performed  by  the  developer.  Government  agencies  may 
witness  these  tests  to  the  extent  possible  on  a  non¬ 
interfering  basis.  The  formal  acceptance  testing  by 
the  developer  will  be  witnessed  by  Government 
representatives.  Chapter  4  describes  the  role  of  the 
T&E  team  members.  While  these  may  be  abbreviated  (or 
sometimes  in  the  case  of  independent  evaluators,  not 
required) ,  the  roles  and  responsibilities  are  similar. 

b.  Developmental  and  operational  testing  brings  a 
variety  of  organizations  into  the  test  process.  The 
test  director's  role  is  carried  out  by  the  assigned 
tester  designated  lAW  AR  73-1.  Testing  will  be 
performed  by  user  personnel  who  have  practical 
operational  experience  in  the  functional  area.  Support 
is  provided  by  the  proponent  and  the  materiel 
developer.  For  specialized  operational  testing  which 
is  required,  testing  may  be  done  solely  by  the  user, 
without  a  test  director,  with  limited  support  from 
other  agencies. 

c.  For  major  changes,  which  generate  issues  and 
criteria  documented  in  the  TEMP,  separate  evaluators 
may  be  designated  lAW  AR  73-1  for  testing  the  change. 
The  functional  proponent  may  be  directly  involved  and 
MACOM  representatives  may  test  and  formally  accept 
changes  for  implementation  within  their  commands. 
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d.  Test  and  evaluation  activities  during  PDSS  need 
to  assure  that  software  meets  requirements,  does  not 
impair  existing  functionality  or  performance,  can  be 
employed  by  the  users,  and  is  effective  and  suitable. 
Chapters  12  through  16  discuss  a  formalized  process  for 
T&E  that  covers  the  major  blocked  changes  for  which 
significant  T&E  is  programmed.  All  changes  must  go 
through  software  development  tests,  a  system 
developmental  test  and  some  form  of  operational 

The  scope  and  significance  of  the  changes 
will  determine  the  formality  of  the  T&E  events,  such  as 
reviews.  In  all  cases  reviews  are  recommended,  but  may 
be  less  structured  and  less  formal  than  those  for  major 
changes.  The  emphasis  is  on  the  types  of  T&E  events 
that  should  occur. 


e.  MSCR  and  AIS  changes  require  consideration  for 
independent  operational  testing  lAW  AR  73-1  and  AR  70- 
15.  The  following  criteria  will  be  used  to  determine 
If  independent  operational  testing  will  be  requiR^efor 
changes  in  computer  resources  (hardware,  software, 
firmware,  communications):  .r.a. 

(1)  Changes  in  computer  resources  which  have  a 
pnysical  impact  on  either  the  operation  or 
supportability  of  the  system. 

(2)  Changes  which  have  a  noticeable  impact  on 
the  system  operational  effectiveness  and  suitability, 
affect  user  interfaces  or  impact  critical  mission 
functions.  Critical  mission  functions  are  those  system 
functions  which  have  a  significant  impact  on  the 
ability  of  the  system  to  perform  a  required  mission,  or 
impact  the  defined  COIC. 

(3)  Changes  that  have  been  made  since  the  system 
was  last  subject  to  independent  operational  testing  and 
which  cumulatively  affect  at  least  15%  of  the  computer 
software  units  of  the  system.  Software  changes  include 
those  which  involve  changes  to  data  relationships  among 
computer  software  units,  content  of  source  or  object 
code,  or  other  relevant  measures. 


The  intensity  of  independent  operational  testing 
which  IS  recommended  by  OPTEC  reflect  the  level  of 
change  to  the  computer  resources.  Recommended  test 
approaches  include  the  use  of  combined  DT  and  OT, 
innovative  operational  tests  and  a  strategic  subset  of 
the  original  test  protocol.  The  objective  is  to  verify 
that  system  operational  effectiveness  and  suitability 
have  not  degraded.  For  system  computer  resources  which 
do  not  meet  the  criteria  in  paragraph  ll-4e,  a  waiver 
request  may  be  submitted  by  the  program  sponsor  or  PEO. 
Waiver  approval  is  required  by  the  user  representative 
(CBTDEV  or  FP)  and  the  system  or  materiel  change 
approval  authority.  Some  of  the  factors  considered  in 
determining  whether  independent  operational  retesting 
IS  necessary  are  shown  in  Figure  ll-i.  some  or  all  of 
these  may  be  affected  by  changes  made  to  software. 
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1.  PunctiOB  (purpoB*)  of  systoa. 
a.  Did  any  functionality  change? 

Crit«l.°“o°c 'ctant,.?’'  Opration.!  .„a 

c.  the  deroonatrated  performance  change? 

change?  *  OMS/MP,  MNS/ORD  or  means  of  employment 

flawV?  potential  for  creating  flaws  or  uncovering 

new  Vunct?ona?  area?®*  •xp-nsion  of  the  system  into  a 

g.  Is  a  degradation  in  performance  anticipated? 
design  throughput*  ‘'*'*"^*  throughput  greater  than  50%  of 

ayetem  survivability  affected? 

J*  **»''«  the  modifications  affected  the  level  of  the 

•thievement  of  the  required 
operational  performance  requirements? 

2*  Testing* 

Should  the  operational  Measure(s)  of  Effectiveness 
(MOE)  or  Measure(s)  of  Performance  (MOP)  change? 

create  a  new  representative  sample 
(modification  causes  the  modified  block  to  become  the  most 
stressful  for  system  performance)? 

tes/pu/Ases?'^  stimulator,  simulator,  or  emulator  required  for 

and  test  and  evaluation  methodologies 

and  histones,  including  requirements  testability,  test  ’ 
completeness,  test  coverage,  adequacy  of  test  cases  and 
procedures,  and  retest  completeness? 

e.  Have  there  been  changes  to  the  Failure  Definition/ 

Scoring  Criteria  (FD/SC)?  tinicion/ 

Interface . 

b*  change? 

D.  IS  tn«re  chsngs  m  systsm  psrformancs  caused  bv 
or  management  of  slave  units  or  network  management?  ^  •’'•^ution 
S’  «  protocols  for  communication  links  affected? 

e’  ill  thSS!  Sh!"®*"  S*'*  output  domains? 

e.  Are  there  changes  in  databus  throughput? 

ss::  tni.  “=’5 

.o(t«r,"in«r*.CM7“^”  <>"  h.rdw.r./ 

4.  Manpower,  personnel,  and  training. 

(»*•>  o(  op.r.tor.  or 

fHAfio'i  \  H.npowor  R.qulr«nont  Critoria 

(MARC),  Basis  Of  Issue  Plan  (BOIP),  or  Qualitative  or 
Quantitative  Personnel  Requirements  Information  (QQPRI)? 

c.  Does  the  o^rator's  manual  or  technical  manual (s)  chanoe? 

d.  Does  the  change  affect  training  drivers  or  systlms?  ® 


Pigur*  11-1.  Indepttndent  Operational  Ratast  Factors 

cases,  a  change  to  one  or  more  of 
retire  the  software  (and  system)  to  be 
retested.  An  example  of  this  is  using  the  system  in  a 
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5.  Support  of  gygtoa. 

a.  la  there  e  change  in  PDSS  maintainability? 

b.  le  there  a  change  in  the  intended  environment 
(particularly  with  reapect  to  temperature  or  vibration? 

c.  Are  there  changes  in  aupport  facilities  (e.g.  space, 
power  requirements)? 

d.  Does  the  change  affect  Built  in  Test  (BIT)  or  Built  in 
Test  Equipment  (BITE)? 

vulnVrabil*  t  V'*  ®*'**'^*  affect  reliability,  survivability,  or 

there  a  change  in  frequency  of  substantial  problems 
during  deployment  ee  a  result  of  a  change  in  reliability# 
survivability#  or  vulnerability? 

g*  Is  configuration  management  adequate  (e.g*  is  the  version 
of  the  software  known;  a  letter  of  certification  from  the  PM)? 

h.  What  is  the  SEE  metric  rating  of  the  system  developer? 

1.  What  is  the  method  for  physical  delivery  and  update  of 

the  software#  and  how  mature  is  the  method? 

6.  Internal  to  system  itself. 

a.  Does  the  change  affect  the  way  the  system  operates? 

b.  Does  the  change  involve  a  change  in  the  architecture  of 
the  system? 

c.  Is  there  a  change  in  throughput  with  respect  to 

or  validation  of  the  compiler? 

d.  Are  there  increased  roemor  assignments  for  the  link-edit 
function? 

e.  Arm  digtributed  procggging  or  databgg*  integrity 

affected?  ’  ■' 

f.  How  many  patches  to  the  software  are  there  in  the  syetem? 

g.  Did  the  modification  include  recompilation  to  integrate 
assembly  language  patches? 

...j'*  "'edifications  involve  tools  or  products  that  were 

different  from  those  used  originally  to  develop  the  system? 

What  was  the  maturity  of  and  what  was  the  software  developer’s 
experience  with  the  tools  and  products  used  to  make  the 
modifications? 

i.  What  is  the  interdependence,  before  and  after  the 
modifications,  among  CSUs,  CSCs,  and  CSCIs? 


Figur*  11-1.  IndBpmndmnt  Opmrationftl  Ratast 
Factors  (Cont'd.) 

manner  that  is  not  the  same  as  the  way  it  was 

originally  designed.  A  change  in  COIC  could  cause  this 
condition. 

11-7,  TtE  considerations  for  anargancy  changes 
a.  When  urgency,  limited  test  requirements,  or 
limited  resources  dictate  that  special  considerations 
be  given  to  a  test  effort,  special  T&E  considerations 

are  required.  Some  special  test  and  evaluation  methods 
are : 

(1)  Combining  developmental  and  operational 
tests.  If  test  objectives  can  be  satisfied  combining 
the  independent  tests,  they  should  be  combined  lAW  AR 
7  3  "  1 . 

(2)  Supplemental  site  testing  (SST) .  When  all 
configurations  of  targeted  hardware  do  not  exist  at  one 
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have  the 

required  environment  may  be  necessary. 

verification  test  (LSVT) .  When 
imited  testing  IS  required,  a  lead  site  may  be 
selected  to  validate  software  changes.  This  method  of 

Changes.  The  selected  test  site  may  be  a  site  that 

software  problem  that  is  being  corr^ted, 
or  »ay  be  any  other  appropriately  selected^site. 

also  is^intInHfH^S"®i  verification  test  (FVT) .  This 
chanoifi  ®  verification  that  quick 

ba«!!;;,rpro«:si"!  •  without  dogrodlng 

(5)  Abbreviated  verification  test.  This  test 
may  be  conducted  by  the  software  developer  with 
appropriate  T&E  community  witnesses  and  may  consist  of 
selected  procedures  end  test  items  that  wilHpecific 

regression  testing  will  also  be 
conducted.  This  type  of  testing  is  used  to  validate 

Change  packages  and  is  usually 
supplemented  by  SST,  LSVT,  or  FVT.  ^ 

chanAoe^JH®i®^®^  testing  will  be  conducted  on  all 
are  to  be  incorporated  into  a  revised 
baseline.  Considering  the  nature  of  the  chanae  and  i-ho 
required  turnaround  time,  the  MATDEV  determines  the 
amount,  extent  and  level  of  developmental  testina 

«  SutliSid%*niSlisly 

c.  The  commander  of  the  MATDEV  activitv  release.? 

Benli!^!''  '='“^°“9h  tbe  clnhlullllll 

d.  As  indicated  in  paragraph  ll-2b  above 

the°nexh^si??  ®"'®^9ency  release  is  required  in 

changir  package  (SCP)  or  plLned  block 

chlaies^**^  regression  testing  for  PD88 

testing  during  PDSS  shall  be 
ensure  adequate  test  coverage  for  the  new 
software  changes  and  adequate  retesting  to  ensure  that- 
fixes  have  not  degraded  existing  functions.  Adeguate 

development  means  that:  (1?  every 

lltt  (tf  addressed  by  at  least  oie  ^ 

test,  (2)  test  cases  have  been  selected  for  both 

••average"  situations  and  "boundary"  situa?ionr  such  as 
^nimum  and  maximum  valu.s,  (3)  "itr.ssS  ««s  hav? 

.  ®”  ®®^®®^®^.  a^ch  as  out-of-bounds  values,  and 

cases  that  exercise  combinations  of  different 

®^®  Retesting,  called  regression 

testing,  as  a  minimum  should  include  repeating  a  sublet 
of  test  cas®s  and  test  procedures  after  sStwLe 

P?evnSr?es?inS  problems  found  in 

p  US  testing.  Retesting  is  complete  if:  (i)  all 
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test  cases  and  test  procedures  that  revealed  problems 
in  the  previous  testing  have  been  repeated,  their 
results  have  been  recorded,  and  the  results  have  met 
acceptance  criteria,  and  (2)  all  test  cases  and  test 
procedures  that  revealed  no  problems  during  previous 
testing,  but  are  test  functions  that  are  affected  by 
the  corrections,  have  been  repeated,  their  results  have 
been  recorded,  and  the  results  have  met  acceptance 
criteria. 

11-9.  System  post-deployment  review  (8PR) 

The  system/operations  manager  or  PM  should  plan  to 
conduct  one  or  more  SPRs  during  PDSS  to  determine  how 
well  the  system  functions.  The  SPR  is  conducted  at 
least  once,  approximately  six  months  after  all  initial 
units  are  equipped  or  all  site  installation  is 
completed.  This  review  should  assess  how  well  the 
operational  system  is  satisfying  user  requirements  in 
relation  to  meeting  the  stated  mission.  In  addition, 
the  review  should  assess  the  degree  to  which  the  system 
operates  as  the  user  understands  or  expects  it  to 
provide.  Results  of  the  SPR  are  used  by  the  MATDEV  to 
ident  rfy  areas  and  changes  that  will  improve  system 
per f 05  nance  and  usability.  Additional  reviews 
throughout  the  deployment  and  operations  phase  provide 
assurance  that  the  system  changes  continue  to  satisfy 
user  needs  and  improve  the  overall  quality.  Content  of 
the  reviews  is  dictated  by  the  initial  system 
corrective  actions,  problem  areas  and  changes. 
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Chaptar  12 

Davalopmant  Dafinition  and  Daaign  During  PD88 
12-1.  Introduction 

Test  and  evaluation  personnel  should  participate  in  all 
software  related  reviews,  such  as  the  Software 
Specification  Reviews  (SSR) ,  Preliminary  Design  Reviews 
(PDR) ,  Critical  Design  Review  (CDR) ,  and  the  Test 
Readiness  Review  (TRR) .  As  during  development,  it  is 
imperative  that  TIE  personnel  understand  what  baselines 
are  changed,  what  software  processing  may  be  impacted 
by  changes,  how  the  changes  will  affect  current 
inputs /outputs  and  finally,  what  expected  results  and 
benefits  derive  from  the  new  changes.  This  information 
better  enables  the  TiE  community  to  develop  plans  and 
procedures  that  thoroughly  test  the  revised  software. 
During  preliminary  design,  focus  is  placed  on  high- 
level  implementation  of  each  requirement.  The  T&E 
community  should  use  this  opportunity  to  formulate 
independent  evaluation  plans,  issues  and  criteria,  test 
cases  and  test  conditions.  At  detailed  design,  the  T&E 
community  gets  the  critical  details  necessary  to 
develop  in-depth  test  procedures  and  test  items.  These 
procedures  or  items  are  used  during  execution  of  formal 
testing  and  the  results  form  the  basis  for  the  test 
report. 

12-2.  Documentation  changes  during  design 
Problems  or  changes  encountered  in  the  design  effort 
may  require  changes  to  the  requirements  documents.  All 
changes  to  baselined  documents  require  formal 
processing  and  approval  within  the  configuration 
management  system.  The  preliminary  updates  should  be 
closely  monitored  to  ensure  the  incorporation  of 
measurable  and  testable  statements,  definitions  and 
criteria.  These  documents  are  further  updated  and 
finalized  during  detailed  design,  resulting  in  revised 
baseline  documents  that  must  be  submitted  for  CCB 
approyal.  The  revised  and  approved  documents 
constitute  the  new  baseline,  which  is  placed  under 
configuration  management.  The  design  reviews  and 
events  described  in  Chapter  6  are  repeated.  The  scope 
and  complexity  of  changes  dictate  how  involved  or 
abbreviated  those  activities  are.  Developers,  SQA, 

IV&V,  software  engineers,  user  representatives,  and  T&E 
personnel  are  encouraged  to  use  informal  action  officer 
walk-throughs  to  work  through  the  technical  and 
functional  aspects  of  the  design  in  relation  to  the 
stated  changes. 

12-3.  Test  and  evaluation  documentation 

a.  T&E  documents  generated  during  PDSS  must  be  the 
result  of: 

(1)  Comprehensive  reviews  of  existing  T&E  plans. 
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(2)  Assessment  of  the  proposed  software  change 
package  for  new  requirements. 

(3)  Assessment  of  the  software  change  package 
for  corrections  to  existing  functions. 

(4)  Review  of  existing  baseline,  test  cases, 
test  scenarios,  files,  job  control  language  ( JCL) , 
benchmark  files,  etc. 

b.  Based  on  comprehensive  reviews  of  the  above 
information,  the  TiE  community  should  update  previous 
existing  test  packages  to  ensure  adequate  test 
procedures  are  generated  to  sufficiently  test  all  new 
requirements  and  corrected  software. 

12-4.  Test  and  evaluation  activities 

The  test  and  evaluation  activities  which  occur  during 
this  phase  are  described  in  Chapter  6.  This  includes 
the  reviews,  audits,  configuration  management 
practices,  metrics  collected  and  analyzed,  user 
demonstrations  to  check  implementation  of  requirements 
in  the  design,  and  entry/exit  objectives.  These 
activities  are  required  by  DODD  5000.1,  DODI  5000.2, 
DODD  7920.1,  DODI  7920.2,  and  AR  73-1.  Continuous 
evaluation,  tracking  of  metrics  and  consistent  T&E 
feedback  provide  the  system/operations  manager  or  PM 
with  the  tools  to  manage  risk  and  make  informed 
decisions. 

12-5.  Responsibilities 

Responsibilities  for  PDSS  test  and  evaluation 
activities  are  provided  in  AR  73-1,  AR  70-15,  and 
AR  25-3. 
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Cbaptar  13 

Softvara  Davalopmant  T&E  Dxiring  P088 
13-1.  Introduction 

a.  The  difference  between  the  PDSS  T&E  process  and 
the  development  TiE  process  is  that  the  primary  focus 
of  PDSS  testing  and  evaluation  is  on  the  specific 
changes  and/or  additions,  along  with  their  effect  on 
the  software  system.  The  activities  involved  during 
coding  through  developer  test  will  be  addressed,  along 
with  the  types  of  documents  required  for  each  activity. 
Depending  upon  the  extent  of  changes,  updates  to 
existing  test  cases,  data  files,  software 
specifications,  etc. ,  efforts  in  the  documentation  area 
concentrate  on  adding  the  changes  and  updates,  not  on 
generating  new  documents. 

b.  In  an  iterative  fashion,  the  design  and  code  are 
developed.  The  related  user  and  recpairements 
specifications,  interface  documents,  design  documents, 
functional  description,  test  plans  (not  procedures)  and 
preliminary  procedures  should  be  updated  and  approved 
lAW  the  stated  CCB  package,  engineering  change 
proposals  (ECPs) ,  and  engineering  change  proposals- 
software  (ECP-Ss). 

13-2.  Coding  and  Unit  (C8D) /module  test 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  coding  and  unit  (CSU) /module 
level  testing. 

b.  Documents.  The  docxments  listed  in  Table  13-1 
are  updated  and/or  developed  during  coding  and  unit 
(CSU)  /module  testing.  Docviments  should  generally 
require  updating  not  complete  development. 


Table  13-1.  Coding  and  Unit  (C8U) /Module  Test 
Doeximents  During  PD88 


Priaary 

MSCR 

Condition 

Deeiaion 

Ratponsibility 

Preduett 

of  Doeuaent 

Alt  Products 

Criteria 

Software 

Developer 

SDFs 

In  Proceaa 

Program  foldera 

None 

STDs 

Updated, 

Software  teat 

Updated, 

new  teat 

caaea /procedurea , 

new  teat 

caaea 

teat  plan  £ 
procedurea 

caaea 

STRs 

In  Proceaa 

Problem  reporta 

None 

c.  Test  activities.  See  Chapter  7,  paragraph  7-2. 

d.  Evaluation  activities.  See  Chapter  7,  paragraph 
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e.  If  applicable,  the  developer  is  responsible  for 
the  SDFs  or  program  folders  and  their  contents. 
Evaluations  of  the  folders  are  usually  conducted  by  an 
independent  organization  or  management  structure 
separate  from  the  software  developer  (i.e.,  usually  the 
Government’s  SQA  organization).  Updates  to  published 
system  documentation  and  training  packages  are  not 
normally  available  at  unit/module  level  testing. 
Documentation  available  should  be  reviewed. 

f.  Metrics.  Metrics  collected  and  analyzed  are 
shown  in  Chapter  7,  Table  7-1.  Detailed  instructions 
for  metrics  collection  and  analysis  are  in  Chapter  17, 
Software  TiE  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

13-3.  CSC/program  integration  and  test 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  CSC/program  level  testing. 
During  this  level  of  testing,  the  developer  integrates 
the  modules  into  programs  or  the  computer  software 
units  into  CSCs.  Unit  level  or  module  level  testino 
must  have  been  successfully  completed  prior  to  CSC 
level  testing.  All  CSCs/programs  should  accept  ’ 
inputs  and  produce  correct  outputs.  Integra*" : 

this  level  needs  to  consider  the  regression  r;. 
factors  described  in  paragraph  11-8  of  Chapter  11. 

b.  Documents.  The  documents  listed  in  Table  13-2 
are  updated  or  in  process  during  CSC/program  level 
testing. 


^•Dle  13-2.  CSC/Program  Integration  Documents 
During  PD88 


Priaary 

MSCR 

Condition 

Decision 

Responsibility 

Products 

of  noetiaant 

AIM  Products 

Criteria 

Software 

Developer 

SDFs 

In  Process 

Program  Folders 

None 

STDs 

Updates 

User  Manual  (UM) 

Updates 

End  User 

Updates 

- 

Manual  (EM) 

STRs 

In  Process 

Problem  Reports 

None 

c.  Test  activities.  See  Chapter  7,  paragraph  7-3. 

^  ^d.  Evaluation  activities.  See  Chapter  7,  paragraph 

e.  Metrics.  Metrics  collected  and  analyzed  are 
shown  in  Chapter  7,  Table  7-3.  Detailed  instructions 
for  metrics  collection  and  analysis  are  in  Chapter  17 
Software  T&E  Metrics,  of  this  part  of  DA  Pamphlet  73-i. 
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13-4.  CSCl/cyela/subsyataB  tast 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  CSCI/cycle/subsystem  level 
tests.  During  this  level  of  testing,  the  developer 
integrates  programs  into  subsystems  or  the  CSCs  into 
CSCIs.  Regression  testing  considerations  apply  as 
described  in  paragraph  11-8  of  Chapter  11.  Program  or 
CSC  level  testing  must  have  been  successfully  completed 
prior  to  CSCI/ subsystem  level  testing. 

b.  Documents.  The  documents  listed  in  Table  13-3 
are  updated  or  in  process  during  CSCI/cycle/subsystem 
level  testing. 


Table  13-3.  CSCI/Cycla/Subsystaa  Doeuaants 
During  PD88 


Priaary 

Rasponsibility 

NSCR  1 

Products  1 

Condition 
of  Docuaont 

AZS  Products 

Decision 

Criteria 

Software 

Developer 

SDFs 

i 

In  Process 

Program  Folders 
(PFs) 

None 

STDs 

Final  Update 

UM 

Updated 

EM 

(if  applicable) 

Updated 

STRs 

In  Process 

Problem  Reports 

None 

Program 
Maint • 
Manual 

Updated, 

Final 

Maintenance 

Manual 

Updated, 

Final 

SPS 

Updated 

Computer 

Operation  Manual 
(if  applicable) 

Updated 

VDD 

Draft 

VDD 

Draft 

Software 

Test 

Report 

Final 

Test  Analysis 
Report 

Final  or  in 
process 
(updated 
when  system 
level  test 
completed) 

c.  Test  activities.  See  Chapter  1,  paragraph  7-4. 

d.  Evaluation  activities.  See  Chapter  7,  paragraph 
7—4  . 

e.  Metrics.  Metrics  collected  and  analyzed  are 
shown  in  Chapter  7,  Table  7-5.  Detailed  instructions 
for  metrics  collection  and  analysis  are  in  Chapter  17, 
Software  T&E  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 
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13-5.  Systam  taat 

a.  Introduction.  This  section  discusses  the 
activities  involved  during  system-level  tests.  During 
this  level  of  testing,  the  developer  completes 
integration  of  software  with  software  and  hardware  to 
form  the  system.  Cycle  or  CSCI  testing  must  have  been 
successfully  completed  prior  to  system  testing. 

b.  Documents.  The  documents  shown  in  Table  13-4 
are  updated  during  system  level  testing. 


Table  13-4.  Dystea  Test  Doeuaents  During  PD88 


Priaary 

Responsibility 

MSCR 

Products 

Condition 
of  Document 

AZ8  Products 

Decision 

Criteria 

Software 

Developer 

SDFs 

In  Process 

Program  Folders 

None 

STDs 

Final 

Updates 

Software  test 
procedures  in  PT 

UK 

(if  applicable) 

EM 

(if  applicable) 

Finsl 

F-iil 

Update 

Final 

Update 

STRs 

In  Process 

PRs 

None 

SPS 

Final 

Update 

Computer 

Operation  Manual 
(if  applicable) 

Final 

Update 

Program 

Final 

Maintenance 

Final 

Maint • 
Manual 

Update 

Manual 

Update 

VDD 

Final 

VDD 

Final 

Software 

Test 

Report 

Final 

Test  Analysis 
Report 

Final 

c.  Test  activities.  See  Chapter  7,  paragraph  7-5. 

d.  Evaluation  activities. 

(1)  System  level  testing  concludes  with  the 
tester  preparing  a  test  (analysis)  report.  The  test 
(analysis)  report  will  be  forwarded  for  review  by  the 
PM  and  the  developmental  and  operational  testers.  This 
final  developer's  test  (analysis)  report  forms  a 
portion  of  the  basis  for  the  system/operations  manager 
to  prepare  the  DTPS  and  conduct  a  DTRR  to  enter  the 
next  test  phase. 

(2)  The  developer  concludes  system  test 
activities  by  updating  the  source  code  and  design 
documentation,  resolving  the  STRs  or  PRs  and  preparing 
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the  test  report.  System  tests  may  be  Government 
witnessed  software  acceptance  tests  to  demonstrate  that 
the  development  testing  has  met  the  software 
specifications  (CCB  package,  ECP-Ss,  etc.),  and  that 
the  changes  have  not  impaired  the  performance  or 
features. _  It  is  particularly  important  for  the 
Government  to  witness  the  system  test  when  the 
development  is  done  by  contractors,  not  Government 
personnel.  Usually  by  contract  the  software  and 
associated  products  are  formally  accepted  by  the 
Government  and  there  must  be  evidence  that  requirements 
are  met  I AW  the  contract . 

e.  Metrics.  Metrics  collected  and  analyzed  are 
shown  in  Chapter  7,  Table  7-7.  Detailed  instructions 
for  metrics  collection  and  analysis  are  in  Chapter  17, 
Software  T&E  Metrics,  of  this  part  of  DA  Pamphlet  73-1. 
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Chap tar  14 

Systam  Davalopmantal  Taat  and  Evaluation  in  PD88 
14-1*  Introduction 

During  this  phase  of  the  process,  the  tester  controls 
the  T&E  process.  The  software  baseline  is  frozen  for 
this  phase  of  testing  in  order  to  maintain  data 
consistency.  The  baseline  is  not  modified  during  this 
phase  of  testing,  unless  severe  problems  are 
encountered,  if  the  baseline  is  modified,  regression 
testing  is  required,  lAW  paragraph  11-8  of  Chapter  11, 
to  ensure  detected  problems  were  corrected  and 
additional  problems  were  not  introduced  into  the 
software.  Test  and  evaluation  objectives,  functions 
and  responsibilities  parallel  those  described  in 
Chapter  8,  System  Developmental  Test  and  Evaluation, 
but  are  usually  abbreviated  based  upon  the  number, 
magnitude  and  complexity  of  modifications.  Excessive 
priority  l,  2,  or  3  software  problems  discovered  during 
testing  may  cause  suspension  or  termination  of  testing. 
Termination  or  suspension  is  lAW  AR  73-1.  The  system 
developmental  test  and  evaluation  described  here  is  the 
same  as  the  PDSS  Test  ,of  Part  Four  of  DA  Pamphlet  73-1. 

14-2.  Entry  objectives 

a.  Evidence  of  successful  completion  of  developer's 
tests,  formal  and  informal,  consists  of  a  test  analysis 
report,  developer  test  docximentation  (e.g.,  updated 
test  plan  and  procedures,  STRs  or  ECP-S  generated),  QA 
configuration  controlled  software  baseline,  baseline 
name  and  version  identifiers.  Preferably,  there  should 
be  no  open  priority  l  or  2  problem  reports  or  incident 
reports  from  previous  testing.  Serious  consideration 
and  severe  test  limitations  may  result  if  DT  occurs 
with  open  priority  l  or  2  problem  reports.  It  may 
cause  DT  to  be  repeated  due  to  invalid  data.  Status  of 
these  are  discussed  at  the  DTRR  (see  Chapter  8) 
conducted  by  the  system/ operations  manager  or  PM. 

b.  Documents  include  an  updated  lEP  if 
independently  evaluated,  either  a  TDP  and  DTP,  or  PT. 
The  system  documentation  (e.g.,  user's  manuals,  updates 
to  VDD,  implementation  instructions,  changes  to 
technical  manuals)  for  the  change  package  will  be  in 
final  form. 

c.  A  DTRS  is  reviewed  and  finalized  during  the 
conduct  of  the  DTRR  to  certify  readiness  to  begin  the 
developmental  testing.  The  DTRR  is  conducted  by  an 
organization  other  than  the  developer.  In  some  cases 
this  may  be  a  QA  organization,  the  PM,  the  operational 
manager,  an  IV&V  agent,  or  other  Government  agency. 
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14-3.  Aetivitias 

a.  System  test 

(1)  DT  verifies  system  performance  to  determine 
if  any  changes  to  the  code  have  introduced  additional 
problems  to  existing  software,  or  performance  has  been 
adversely  J.mpacted.  Additionally,  volume  loads,  stress 
levels  and  interfaces,  and  system  interoperability  and 
interfaces  are  tested.  Benchmark  test  files,  copies  or 
production  files  and  test  cases  entered  by  users 
constitute  the  test  data  based  upon  the  ECP-S  or  other 
change  package.  Testing  will  not  be  run  in  a  live 
production  environment,  but  will  be  run  in  parallel. 

If  this  is  not  feasible,  a  waiver  must  be  obtained  from 
the  approval  authority  for  that  level  of  system. 

(2)  The  tester  will  coordinate  all  activities 
with  the  PM  and  provide  guidance  for  the  resources 
required  to  support  testing.  At  the  beginning  of  DT 
activities,  updates  to  computer  operation  and  user's 
manuals,  conversion  documentation,  VDD  and  training 
package  materials  will  be  in  final  form  and  will  be 
used  in  the  execution  of  the  DT  processes. 

(3)  The  tester  updates  existing  test  plans  and 
test  cases  based  upon  the  ECP-S  to  ensure  that  tl 
existing  functions  work  and  new  requirements  are 
stated. 

(4)  Well-defined  requirements  are  necessary  to 
develop  the  test  plan  for  all  evaluation  issues.  These 
requirements  will  allow  testers  to  tailor  the  generic 
test  areas  to  the  specific  software  application. 

(5)  If  an  independent  developmental  evaluator  is 
designated,  he  shall  monitor  the  test,  but  not 
participate  as  a  test  director.  The  developer  is  not 
to  participate  in  this,  unless  requested  by  the  tester 
when  difficulties  occur. 

(6)  Problem  reports  are  generated  for 
system/software  anomalies  found  during  DT.  Problems 
are  reported  in  a  TIR,  PR,  or  STR.  Software  problems 
are  categorized  and  prioritized  lAW  DOD-STD-2167A, 
Appendix  C.  Problem  report  forms  often  have  a 
developer's  analysis  section  which  is  used  for 
corrective  action  reporting. 

b.  Evaluation 

(1)  Results  of  testing  are  scored  by  a  review 
board.  The  user  representative,  developmental 
evaluator  if  applicable,  and  developmental  tester  are 
principal  board  members.  Test  results  are  evaluated  by 
the  developmental  evaluator/assessor  or  tester  lAW  the 
test  plan. 

(2)  Tests  are  structured  to  ensure  that  all 
capabilities  and  requirements  of  the  system 
modifications  are  exercised  and  analyzed  lAW  the 
requirements  stated  in  the  ECP-S  and  other  documents. 
Elements  of  the  developmental  evaluation/assessment  may 
include  but  are  not  limited  to  the  following: 
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well  performance  —  determine  how 

e®  system  performance. 

Examples  of  perfonnance  issues  are: 

”  Control  statements  adequately  reflect  the 
required  functional  order  of  cycle  and 
system  processing  (e.g.,  files  passed 
between  cycles) . 

-  Performance  results  are  evaluated  to  ensure 
that  response  time,  run  times,  memory  and 
peripheral  equipment  utilization,  operator 
intervention  requirements,  and  overall 
operating  characteristics  are  analyzed. 

This  information  will  be  used  to  determine 
potential  adverse  system  impacts  and 
possible  performance  bottleneclcs  as  well  as 
significant  benefits. 

-  Recovery/restart  procedures  are .evaluated 
to  ensure  that  users  can  overcome  potential 
processing  malfunctions. 

-  Test  results  are  fully  evaluated  to  ensure 
that  all  stated  test  objectives  have  been 
met,  or  that  sufficient  explanation  is 
given  for  those  objectives  not  attained  and 
the  impact  on  the  OT  or  other  post-DT 
activities. 

“  The  conversion  process  is  evaluated  to 
ensure  that  data  is  transferred  in  a 

manner  (e.g.,  left  of  baseline).  A 
similar  evaluation  must  also  be  performed 
for  processing  which  generates  the 
converted  or  initial  database  into  the  new 
system  format  (e.g.,  right  of  baseline). 

®®*'t®nts  resulting  from  the  conversion 
process  must  be  verified  and  validated 
according  to  the  methodology  and  criteria 
specified  in  the  DT  test  plan. 

,  .  Interoperability  —  the  degree  to  which 

data  is  correctly  exchanged  and  interpreted  between 
systems.  Basically,  there  are  two  types  of 

intrasystem  and  intersystem. 

Examples  of  interoperability  issues  are: 

-  The  system  user  is  able  to  manage  the 
system  to  include  interactive  terminal 
Interface,  cycle/system  set— up,  and 
input/output  control. 

-  Interface  considerations  with  respect  to 
ease  of  data  handling  through  cycle 
processing,  intersystem  data  transfer, 
transmission  of  data  over  communications 
linJcs,  and  time  sharing  lin)cs  are 
functioning  properly. 

w  Usability  —  The  effort  required  to  learn 

the  human  interface  with  the  software,  to  prepare  input 
and  to  interpret  output  of  the  software.  Usability  ^ 
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testing  addresses  the  nan-machine  interface.  Examples 
of  usability  issues  are:  system  products  such  as 
terminal  displays,  hard  copy  reports,  punched  cards, 
and  magnetic  tape  and  direct  access  files  are  correct 
and  disposition  for  these  are  clear  and  adequate. 

fd)  Training  is  evaluated  to  ensure  the 
adequacy  of  appropriate  user  manuals,  the  VDD,  on-the- 
job  instruction,  and  problem  reporting  procedures,  if 
applicable. 

(e)  Documentation  is  evaluated  to  ensure  that 
appropriate  personnel  can  comprehend  and  use  these 
publications.  The  programmer's  maintenance  manuals 
should  be  updated  and  reviewed.  Special  attention 
should  be  given  to  those  docximents  destined  to  be  used 
by  non-automated  data  processing  (ADP) ,  functionally 
oriented  personnel  in  the  operation  and  use  of  computer 
software  systems. 

(3)  The  developmental  test  report  is  written  by 
the  developmental  tester. 

14-4.  Decision  criteria 

a.  The  items  needed  to  proceed  from  this  phase  and 
enter  operational  testing  .re  an  lER,  if  required, 
developmental  test  (analysis)  report,  and  successful 
completion  of  the  OTRR.  If  independently  evaluated, 
all  independently  evaluated  systems  produce  an  lER  in 
addition  to  a  test  report.  Independent  evaluation  is 
less  frequent  during  PDSS.  The  lER  is  the  evaluation 
of  how  well  the  system  met  the  critical  issues  stated 
in  the  lEP.  This  evaluation  should  include  software 
metrics  and  any  other  outstanding  software  problems. 

For  systems  which  are  not  independently  evaluated  or 
assessed  during  PDSS,  the  tester's  report  shall  include 
evaluative  content. 

b.  For  entry  to  operational  testing,  no  priority  l 
or  2  software  problem  reports  can  be  open.  In 
addition,  software  problems  fixed  during  DT  require 
evidence  of  comprehensive  regression  testing  prior  to 
admittance  for  operational  testing.  This  includes 
updates  to  documentation  and  impacts  of  any  work¬ 
arounds  required.  The  user  may  require  that  additional 
software  problems  be  fixed  prior  to  OT. 

c.  Metrics  collected  during  Government 
developmental  testing  will  include  reliability.  Fault 
discovery  and  closure  data  will  be  collected  and 
analyzed  lAW  Chapter  17,  Software  TiE  Metrics,  of  this 
part  of  DA  Pamphlet  73-1. 

d.  Exiting  this  phase  is  contingent  on  the 
responsible  organizations  and  cognizant  personnel 
conditionally  approving  moving  to  the  next  phase (s). 
This  warrants  that  the  documents,  metrics,  reviews, 
results,  regression  tests,  evaluations  (if  required), 
etc.,  are  in  the  condition  to  be  used  to  execute  the 
next  phase. 
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Cbaptar  15  . 

Oparational  Tasting  (OT)  in  PD88 

15-1.  Purposa 

a.  This  chapter  defines  methodology  and  procedures 
for  operational  evaluation  and  test  implementation  for 
systems  which  contain  computer  resources,  during  the 
post-deployment  phase  of  the  life  cycle.  Operational 
testing  focuses  on  how  the  software  supports  the  system 
mission,  and  the  capability  of  the  user  to  employ  the 
system.  If  new  COICs  are  generated,  the  system 
requires  independent  operational  evaluation  lAW  AR  73- 
1.  In  these  cases,  software  evaluation  aims  for 
successful  operational  testing  in  the  operational 
configurations,  loads,  and  environments;  and  assists  in 
deteraining  adequate  test  coverage  so  evaluators  are 
confident  in  their  assessments  of  operational 
effectiveness. 

b.  Operational  testing  is  distinct  from  other 
tests.  See  Chapter  9,  paragraph  9-1.  Requirements  for 
OPTEC  independent  operational  testing  are  lAW  paragraph 
11-6  of  Chapter  11  and  AR  73-1. 

15-2.  Objective 

The  objective  of  operational  testing  and  evaluation  is 
to  evaluate  the  extent  to  which  the  software  supports, 
and  can  continue  to  support,  the  operational 
requirements  of  the  user.  The  goal  of  PDSS  is  to 
ensure  systems  remain  operationally  effective  and 
suitable. 

15-3.  Operational  testing  types 

a.  Operational  tests  and  their  associated 
evaluations,  if  required  lAW  AR  73-1  and  AR  25-3,  are 
crucial  elements  of  the  decision-maker's  milestone 
process  for  decisions.  OT,  under  specific  conditions 
outlined  in  AR  73-1,  may  be  combined  with  developmental 
testing.  Combined  DT/OT  requires  that  the  T&E  team 
early-on  establish  their  test  strategies  and  coordinate 
their  efforts.  In  all, cases,  a  portion  of  DT/OT  must 
be  a  dedicated  OT  phase.  OT  during  PDSS  is  generally 
an  EOT  and  may  require  independent  evaluation. 

•  b.  Depending  on  the  circumstances  involved,  any  one 
of  the  types  of  operational  tests  listed  in  Chapter  2, 
Table  2-1,  may  be  appropriate  during  PDSS.  The  most 
common  type  is  FOT  which  will  be  used  for  most  software 
change  packages  (SCPs)  or  (ECP-Ss) .  Supplemental 
operational  tests  will  be  used  when  the  system  under 
test  is  deployed  on  more  than  one  hardware  platform,  or 
when  concepts  and  innovative  approaches  will  be 
explored.  Limited  user  tests  (LUTs)  may  be  used  when  a 
priority  1  or  2  ECP-S  is  fielded  in  an  interim  change 
package  (ICP) ,  or  for  any  emergency  change  or  date- 
driven  regulatory  change. 
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15-4.  Entry  objaetivas 

a.  Operational  test  readiness  statement.  The  PM  or 
system/operations  manager,  trainer,  and  the  user 
representative  prepare  OTRSs  to  indicate  that  the  area 
for  which  each  is  responsible  is  prepared  to  enter  the 
OT. 

b.  Operational  test  readiness  assessment.  See 
Chapter  9,  paragraph  9-4. 

c.  Completed  software  technical  and  developmental 
tests,  as  determined  by  the  breadth  of  testing  metric. 
The  tester  or  evaluator,  if  an  evaluated  system, 
ensures  Government  developmental  testing  and  software 
development  aire  completed.  See  Chapter  9,  paragraph  9- 

d.  OT  version  description.  The  operational  tester 
or  evaluator  may  include,  as  part  of  the  OT  readiness 
assessment,  the  version  description  document  (VDD)  or 
similar  description  of  the  version  of  software  to  be 
tested  in  operational  test.  See  Chapter  9,  paragraph 
9-4. 

e.  Software  problem  reports.  Priority  1  and  2 
software  problem  reports  must  be  resolved  prior  to  OT. 
The  system  may  be  allowed  to  proceed  to  OT  with  open 
problem  reports  (priorities  other  than  1  or  2)  provided 
that  concurrence  is  obtained  from  the  user 
representatives.  Work-arounds  may  be  necessary  for 
these  open  problem  reports  in  order  to  avoid  delaying 
the  start  of  OT.  However,  these  must  be  clearly 
specified,  trained  for,  and  accepted  by  the  user 
representative. 

f.  Operational  test  readiness  review.  See  Chapter 
9,  paragraph  9-4  and  Part  Five  of  DA  Pamphlet  73-i. 

15-5.  Documentation 

a.  The  test  plans,  evaluation  plans  (only  if 
evaluated) ,  and  supporting  plans  must  be  updated  and 
approved  and  ready  to  support  the  execution  of 
operational  testing  and  evaluation. 

b.  For  major  and  non-major  systems  (except  AIS 

Class  VI  and  some  AIS  Class  V  lAW  DISC4) ,  the  following 
plans  are  used  and  are  updated  for  execution  during 
this  T&E  event.  The  document  prepared/updated  by  the 
operational  evaluator  and/or  operational  tester  is  the 
test  and  evaluation  plan  (Chapters  1  and  2  of  the  TEP) . 
The  documents  prepared  by  the  operational  test 
organization  are:  the  test  plan  (Chapter  3  of  the  TEP) 
and  the  detailed  test  planning  document  (detailed  test 
plan  (DTP)  or  test  plan  (PT)).  The  scope  and  volume  of 
documentation  for  OT  during  PDSS  may  be  proportionately 
reduced  from  that  required  during  the  development  phase 
of  the  life  cycle.  r' 

c.  Operational  testing,  including  test  planning  and 
test  reporting,  are  still  performed  for  systems  that 

required  to  conduct  independent  evaluations. 
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Documentation  is  less  extensive,  but  the  requirement  to 
perform  adequate  testing  remains  constant.  MACOM-level 
approval  systems  or  installation  approval  systems,  AIS 
Class  VI  and  some  AIS  Class  V  lAW  DISC4,  must  conduct 
operational  testing  to  an  established  set  of  user 
requirements  in  the  operational  environment  just  as  do 
the  higher  approval  level  systems.  Documentation 
prepared  by  the  operational  tester  consists,  at  a 
minimum,  of  the  TEP,  DTP  or  test  plan  (PT) ,  and  the 
test  analysis  report  or  test  report. 

d.  Operational  tests  will  use  final  documentation: 
VDD,  user's  manuals,  end  user's  manuals  and/or  computer 
operation  manuals,  training  documents,  technical 
manuals,  COOP  or  CONOPS. 

15-6.  Activities 

a.  Test. 

(1)  OT  is  a  system-level  test  performed  by  a 
test  activity  independent  of  the  developer,  the  PM, 
system/ operations  manager,  or  the  user  community.  This 
means  that  the  test  planning  scenarios,  etc.,  are 
developed  by  an  independent  activity.  It  is  imperative 
that  the  user  representatives  participate  in  certain 
planning  aspects  and  the  test.  Characteristically, 
the  test  is  executed  in  a  realistic  or  live  environment 
representative  of  the  operational  system  mission,  by 
the  designated  system  users.  The  OT  is  conducted  on 
the  specified  target  hardware,  in  the  production- 
representative  system  configuration  (e.g., 
communications,  peripherals,  other  system  interfaces, 
etc.)  with  the  software  version  to  be  fielded. 

(2)  The  objective  of  OT  is  not  to  test  the 
technical  aspects  of  the  system,  but  rather  the 
entirety  of  the  system  and  its  planned  support 
environment  when  it  is  deployed.  The  structure  of  the 
test  considers  elements  of  effectiveness  and 
suitability  (e.g.,  training,  logistics,  human  factors). 

(3)  Test  requirements  of  the  test  plan  will  be 
exercised  and  satisfied  prior  to  test  tezBinaition. 

(4)  Software  change  policy  during  OT.  Software 
found  to  be  deficient  to  the  extent  that  the  objectives 
of  the  OT  cannot  be  met,  will  require  a  special  OTRR  to 
review  the  extent  of  the  modifications  required,  the 
impact  of  the  modifications  on  the  ability  of  the  test 
to  meet  its  objectives,  and  the  resource  implications 
if  the  completed  portion  of  the  test  has  to  be 
repeated.  Decisions  to  ma)ce  changes  to  software  during 
FOT  or  other  post-deployment  tests  are  the  decision  of 
the  independent  tester;  however  regression  testing  and 
additional  testing  will  be  conducted  to  ensure  that  the 
change  wor)cs  and  has  not  impaired  other  functions. 

(5)  Operational  testing  shall  occur  in  parallel 
with  the  existing  system.  If  this  is  not  feasible,  a 
written  waiver  will  be  prepared  by  the  PM  or 
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system/operations  manager  and  the  operational  tester, 
and  submitted  to  the  test  site  or  local  director  of 
information  management  (DOIM) .  Following  operational 
tests,  the  system/ software  is  removed  from  the  test 
site  and  released  only  through  formal  configuration 
management  release  channels  unless  the  test  waiver  is 
approved;  however,  systems  or  software  cannot  be 
released  to  any  other  sites.  Other  sites  must  wait  for 
release  through  fonnal  channels. 

b.  Evaluation.  If  this  PDSS  change  is  not 
independently  evaluated,  the  FP,  CBTDEV,  or  his 
designated  representative  shall  ensure  the  issues  of 
effectiveness  and  suitability  are  addressed.  The  FP  or 
CBTDEV  shall  then  prepare  the  evaluative  comments 
portion  of  the  expanded  test  report  lAW  AR  73-1. 

Should  the  FP  or  CBTDEV  elect  to  do  so,  the  operational 
tester  shall  prepare  an  expanded  test  report  lAW  AR  73- 
1  with  evaluative  content.  Operational  test  issues  are 
discussed  in  Chapter  9,  paragraph  9-6. 

c.  Metrics.  The  metric  collected  and  analyzed  for 
OT  is  reliability.  Fault  discovery  and  closure  data 
are  analyzed  using  the  software  problem  priorities 
listed  in  DOD-STD-2167A,  Appendix  C.  The  reliability 
metric  is  discussed  in  Chapter  17,  Software  T&E 
Metrics,  of  this  part  of  DA  Pamphlet  73-1. 

15-7.  Decision  criteria 

a.  In  order  for  the  system  to  successfully  complete 
OT,  the  following  documentation  must  be  complete:  an 
approved  TER  or  OA  or  expanded  test  report  (ETR)  lAW  AR 
"73—1.  ^  The  TER/OA  or  ETR  is  based  on  the  test  results 
contained  in  the  test  report  or  expanded  test  report, 
and  any  other  documentation  supporting  the  evaluation 
of  system  or  software  change  package  (SCP) 
effectiveness  and  suitability.  It  identifies  how  well 
the  system  met  the  COIC  or  other  Issues.  If  all  COICs 
have  been  addressed  satisfactorily,  a  recommended 
position  for  materiel  release  is  included  as  part  of 
the  TER/OA  or  ETR. 

b.  Evidence  of  failure  to  meet  the  user  operational 
issues  and  criteria  will  be  the  basis  for  a  decision  by 
the  appropriate  decision  authority.  Among  the  possible 
alternative  courses  of  action  are: 

(1)  To  implement  follow-on  formal  operational 
test(s).  The  emphasis  behind  these  follow-on  tests 
will  be  to  address  the  problems  identified  during 
initial  operational  testing. 

(2)  To  return  the  program  to  the  DT  phase  for 
significant  corrective  action. 

c.  If  follow-on  testing  is  required,  the 
process  for  development  of  T&E  plans  previously 
discussed  must  be  re-initiated  to  address  only  those 
problem  areas  in  the  TER/OA  or  ETR.  At  the  completion 
of  follow-on  operational  test,  the  decision  authority 
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will  make  a  materiel  release  decision  based  upon  all 
information  available  to  that  body  at  the  time,  to 
include  the  recommendation  of  the  evaluator  or  tester 
if  required.  ' 
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Cbaptar  16 

Matarial  Ra-ralaaaa  for  Softvara 
16-1*  Introduction 

As  stated  in  Chapter  10,  Materiel  Release  for  Software, 
the  materiel  release  process  is  intended  to  assure  that 
Army  materiel  is  suitable  and  supportable  before 
release  for  issue  to  the  users.  The  materiel  fielding 
and  transfer  processes  apply  ecpially  to 
modified/upgraded  items  as  they  do  to  new  items. 
Particular  attention  must  be  given  to  modified/upgraded 
items,  because  very  often  they  are  implemented  and 
delivered  to. the  field  without  having  gone  through  the 
proper  assessment  process  -  this  is  particularly 
critical  in  the  case  of  software,  since  a  small, 
misplaced  error  can  cause  major  degradation  of  mission 
effectiveness  and  significantly  impact  safety.  AR  700- 
142  documents  the  Army's  materiel  release,  fielding, 
and  transfer  processes.  This  chapter  emphasizes  and 
expands  upon  the  software  issues  as  they  pertain  to  the 
release  and  fielding  process  for  modified/upgraded 
items  and  emergency  materiel  releases  for  software. 

16-2.  Objectives 

The  objectives  of  the  materiel  release  for  issue 
process  remain  the  same  during  the  post-deployment 
phases;  these  are  to: 

a.  Establish  a  management  control  system  to  ensure 
that  materiel  release  for  issue  by  the  Army  is  safe, 
operates  as  designated,  and  is  logistically 
supportable. 

b.  Provide  a  system  which  enables  HQDA  to  have 
overall  visibility  and  control  of  the  materiel  release 
process. 

c.  Provide  a  mechanism  to  monitor,  control  and 
follow  through  on  all  conditional  releases  until  full 
release  is  obtained. 

16-3.  Scope 

a.  Materiel  subject  to  release  actions  under  AR 
700-142  are  the  same  as  those  stated  in  Chapter  10, 
Materiel  Release  for  Software.  This  section  deals 
specifically  with  changed  software  which  results  from 
actions  to  modify/upgrade  a  fielded  system.  Changed 
software,  includes  embedded,  proprietary,  and  NDI 
software  that  meet  one  or  more  of  the  following 
criteria: 

(1)  Software  that  significantly  changes  (or  has 
the  potential  to  change  if  not  adequately  tested) 
mission  function,  capability,  performance  parameters, 
interoperability  requirements,  reliability  or  safety. 

(2)  A  blocic  update  consisting  of  a  software 
change  of  more  than  30%  executable  LOC  or  30% 
executable  cumulative  LOC  changes  not  having  required 
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release  approval  since  the  last  materiel  release. 

These  criteria  may  be  tightened  at  the  discretion  of 
the  MATDEV  based  upon  the  criticality  of  the  software 
changes . 

(3)  A  block  update  consisting  of  a  software 
translation  of  30%  LOC  to  a  different  computer 
programming  language. 

(4)  Deployable  system  integration  software, 
which  is  newly  developed  software  added  to  NDI  software 
necessary  to  allow  customization  and  interoperability, 
or  simply  to  integrate  multiple  NDI  software  items  so 
that  they  work  together  under  one  "system." 

(5)  Software  that  is  significantly  changed  to 
run  on  a  different  computer  processor. 

(6)  Software  changes  that  require  new  user-level 

test  equipment  and/or  impact  25%  of  the  program  of 
instruction.  .  . 

b.  The  process  as  discussed  herein  applies  to  the 
materiel  discussed  in  paragraph  16-3a  above,  with  the 
following  special  provisions  noted: 

(1)  Materiel  developed  by  the  Army,  procured  by 
the  D^,  and  distributed  by  the  Army  requires  a 
materiel  release  action. 

(2)  Foi"  materiel  developed  by  the  Army  and 
assigned  to  the  DLA  or  the  GSA  for  procurement  and 
distribution  (integrated  materiel  management),  the  Army 
MATDEV  will  establish  an  MOA.  All  necessary  technical 
and  logistic  support  will  be  provided  to  DLA  or  GSA  to 
assure  distribution  of  the  materiel. 

(3)  For  materiel  developed  by  the  Army  for 
another  service,  federal  agency,  or  foreign  government, 
the  criteria  specified  in  the  agreements  between  the 
Army  and  the  user  or  developer  will  govern  the  materiel 
release  process.  To  the  extent  that  those  criteria  are 
not  defined,  the  criteria  in  AR  700-142  apply. 

(4)  Security  assistance  programs  may  be  waived 
from  the  instruction  specified  herein  to  accommodate 
the  terms  of  an  agreement  or  when  requested  by  the 
customer . 


16-4.  Materiel  release  policy 

a.  The  provisions  stated  in  Chapter  10,  Materiel 
Release  for  Software,  remain  in  force  during  the  post¬ 
deployment  phase.  This  paragraph  addresses  additional 
procedures  which  are  implemented  at  MATDEV  MSCs  to  deal 
with  software  upgrades/modifications  implemented  via 
ECPs.  The  rationale  for  these  procedures  stems  from 
repeated  problems  typically  observed  when  software  has 
been  modified  and  issued  to  the  field. 

b.  An  adyisory  group  to  the  Materiel  Release  Review 
Board,  sometimes  referred  to  as  the  MRRB  Software  Sub¬ 
group  (SSG) ,  can  add  significant  emphasis  on  assurina 
the  integrity  of  the  ECP  process.  Made  up  of 
experienced  software  managers,  the  members  of  the  SSG 
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are  fully  qualified  to  address  all  software  issues,  and 
are  representatives  from  all  of  the  cognizant  MATDEV 
organizations  including  software  quality  assurance, 
software  engineering,  safety,  and  ILS.  Depending  on 
the  release  action  to  be  taken,  full  release, 
conditional  re-release  or  simply  issue  via  ECP,  the  SSG 
can  advise  the  MRRB  or  the  configuration  manager 
accordingly.  Therefore,  the  SSG's  charter  fully 
supports  the  activities  of  the  CM  and  can  be  an 
integral  part  of  the  configuration  management  process. 

16-5.  Types  of  aateriel  re-release  during  post¬ 
deployment 

Re-release  of  software  to  the  field  can  be  accomplished 
through  the  materiel  release  process  (as  full, 
conditional,  training,  or  emergency  releases),  based  on 
the  criteria  stated  in  paragraph  16-3.  If  the  extent 
of  the  changes,  modifications,  or  upgrades  to  the 
software  do  hot  fall  under  these  criteria,  then 
typically  the  software  is  released  via  ECP  and 
configuration  management  channels.  In  this  case,  no 
action  may  be  necessary  outside  of  the  normal  ECP 
process;  however,  depending  on  a  less  stringent  set  of 
criteria,  the  ECP  may  need  to  be  accompanied  by  a 
suitability  for  issue  statement. 

a.  Full  re-release.  A  full  re-release  is 
authorized  after  the  materiel  has  been  tested  and 
evaluated,  and  determined  to  meet  all  established 
requirements.  The  user  MACOM  must  also  concur  with  the 
final  materiel  fielding  plan,  and  all  other  aspects  of 
the  logistics  support  system  must  have  been  achieved  as 
specified  in  the  ILSP  approved  at  the  Milestone  III 
decision  review.  In  addition,  validated  and  verified 
draft  DA  equipment  publication  (or  authenticated 
commercial  manuals) ,  and  all  classes  of  supply  and 
sustainment  materiel  must  be  available  prior  to  or 
concurrent  with  fielding. 

b.  Conditional  re-release.  A  conditional  re- 
release  may  be  authorized  when  one  or  more  of  the 
criteria  for  full  release  have  not  been  met.  If  an 
urgent  need  exists  for  the  materiel,  a  conditional  re- 
release  may  be  granted  as  long  as  the  hardware  and/or 
software  is  available  and  acceptable  to  the  user  MACOM, 
and  if  an  interim  means  of  support  exists.  However,  a 
conditional  re-release  requires  that  a  get-well  plan 
(for  each  condition  that  precluded  a  full  re-release) 
be  developed  and  approved,  and  that  the  conditional  re- 
release  be  restricted  to  a  specific  quantity,  location, 
and  application.  Requests  for  conditional  re-release 
of  DOD  major  and  DAP  materiel  systems  are  sent  through 
HQDA  (DALO-SMS)  to  the  VCSA.  Status  must  be  reported 
periodically,  until  all  conditions  are  satisfied  and 
full  re-release  is  realized. 
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c.  Training  re-release.  Training  re-release  is  the 
release  of  materiel  for  training  use  only.  Prior  to 
approving  training  re-releases,  the  MATDEV  ensures  that 
critical  issues  such  as  safety,  availability  of 
spare/repair  parts,  technical  documentation, 
responsibility  for  maintenance  support,  and  other 
conditions  which  limit  use  of  the  item  are  identified 
and  accepted  by  the  trainer. 

d.  Emergency  materiel  releases  for  software  may  be 
required  to  correct  serious  software  problems,  respond 
to  threat  changes,  or  changing  doctrine  during  wartime. 
The  MSC  commander  can  approve  such  fast  turn-around 
releases.  Evidence  of  testing,  especially 
retesting/regression,  is  a  necessary  condition  for 
release. 

e.  Suitability  for  issue  release.  The  suitability 
for  issue  release  is  a  streamlined  attempt  at 
addressing  all  of  the  critical  issues  necessary  for 
materiel  re-release,  while  avoiding  the  need  to  invoke 
the  normal  materiel  release/re-release  process.  This 
is  a  considerable  time-saver  for  software  changes  that 
are  minor  or  considered  "less  critical,”  yet  still 
provides  the  necessary  assurance  that  the  changes  have 
been  properly  evaluated  and  tested.  The  suitability 
for  issue  release  consists  of  a  short,  concise 
suitability  statement  and  all  supporting  data.  What 
follows  is  a  set  of  criteria  which  can  be  implemented 
at  the  MACOM  level  as  supplements  to  AR  700-142,  to 
assist  in  determining  when  a  suitability  for  issue 
release  is  required  when  normal  materiel  re-release  is 
not.  If  the  nature  of  the  change  is  so  minor  that  it 
does  not  even  meet  the  criteria  listed  below  for  a 
suitability  for  issue  release,  then  re-release  is 
handled  strictly  via  the  ECP  process.  The  criteria  for 
suitability  for  issue  release  are: 

(1)  Change  to  data  base  to  incorporate  new  or 
updated  information  (e.g.,  ballistic  data,  signatures, 
fingerprints,  etc.). 

(2)  Change  which  may  have  significant  impact  on 
ILS  (e.g.,  spare  parts,  manuals,  support  equipment, 
etc . ) . 

(3)  Minor  changes  to  mission  function, 
capability,  performance,  interoperability,  reliability, 
safety,  etc. 

(4)  Minor  impacts  to  TRADOC  program  of 
instruction. 

(5)  Other  ECP  actions  (reviewed  on  a  case-by- 
case  basis) . 

(a)  Block  updates. 

(b)  Translation  of  language. 

16-6.  Procedures  for  materiel  re-release 

Identification  of  ECP  actions  involving  software  can  be 
flagged  in  a  number  of  ways,  but  typically  the 
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cognizant  software  quality  assurance  and  software 
engineering  organizations  identify  ECPs.  They  also 
perform  the  initial  evaluation  of  the  scope  of  the 
software  change  against  the  criteria  in  paragraph  16-3 
for  materiel  re-release,  and  the  criteria  in  paragraph 
16-5  for  suitability  for  issue  release.  If  none  of  the 
criteria  are  met,  then  no  release  action  is  necessary 
beyond  the  normal  ECP  configuration  management  process. 
However,  if  the  scope  of  the  software  change  meets  any 
of  the  established  criteria,  the  cognizant 
organizations  are  responsible  for  making  the 
recommendation  for  which  release  action  is  necessary. 
This  recommendation  is  sent  by  memorandum  to  the  SSG 
chairman  who  convenes  the  SSG  for  final  determination. 
This  reduces  the  burden  on  the  SSG  to  constantly  look 
out  for  software  ECP  actions,  and  keeps  their 
involvement  at  a  minimum.  Regardless  of  which  release 
action  is  necessary,  the  SSG  is  responsible  for 
preparing  a  suitability  statement  and  collecting  all 
supporting  data.  Depending  on  which  criteria  were  met, 
the  SSG  sends  the  suitability  statement  to  the  MRRB 
(for  materiel  re-release  per  Chapter  10,  Materiel 
Release  for  Software)  or  to  the  CCB  (for  release  via 
ECP  process  as  a  suitability  for  issue  release.  In  any 
case,  the  suitability  statement  and  support  data  always 
accompany  the  appropriate  release/ fielding  documents. 
Figure  16-1  provides  a  graphical  representation  of  this 
process. 

16-7.  Doevuneatatioa  for  materiel  re-release 
To  ensure  that  the  objectives  of  the  materiel  re- 
release  process  are  met,  strict  documentation  must  be 
prepared  and  submitted  for  release  approval.  AR  700- 
142  details  the  scope  and  preparation,  but  as  a  minimum 
the  materiel  release  documentation  should  include 
independent  evaluations/assessments  from  the 
developmental  and  operational  evaluators/assessors, 
confirmation  of  the  safety  assessment,  statements  of 
supportability  (for  each  hardware /software  major 
constituent) ,  confirmation  that  all  aspects  of  the 
logistics  support  system  plan  agreed  to  at  the 
Milestone  III  decision  have  been  achieved,  and  a  signed 
materiel  fielding  agreement.  Of  significant  note  is 
the  requirement  for  documentation  of  all  software 
supporting  data,  especially  for  re-releases 
necessitated  by  software  modifications/upgrades.  If 
the  materiel  release  is  conditional,  then  a  statement 
is  also  included  from  the  MACOM  user  accepting  all 
conditions  for  release,  along  with  general  officer 
concurrence  from  the  gaining  command. 

16-8.  8pecial  considerations 

a.  The  independent  tester  and  evaluator  functions 
do  not  end  with  fielding/release  of  a  software- 
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intensive  system.  The  independent  evaluators  must 
monitor  production  error  reports  to  identify  required 
modification  to  testbeds.  Evaluation  continues 
through  post-deployment  review  or  operational 
assessment.  Often  this  evaluation  will  become  an 
integral  part  of  the  mission  need  determination  phase 
for  re-design  or  replacement  projects. 

b.  Ideally,  independent  evaluators  stay  with  a 
system  throughout  deployment  and  operations.  In 
reality,  these  functions  are  normally  passed  from  a 
full-time  evaluation  agency  to  the  system/operations 
manager,  functional  proponent,  or  MATDEV. 
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Cbaptar  17 

Softwara  tse  Matrlca 
SaetioB  I 

RacoBaandad  Matric  Sat 
17-1.  ZBtroduetloB 

a.  Per  AR  73-1,  "Software  T&E  must  provide  data  to 
support  an  established  set  of  qualitative  aid 
quantitative  software  oetrics.  These  netrics  serve  as 

chariJ^t^  that  critical  technical 

operational  issues  of  both  software 
metrfoc  achieved."  The 

set  fi?  comprise  a  minimum 

b.  Many  characteristics  of  software  need  to  ho 
evaluated  for  quality  and  conformance  but  do  not  lend 
themselves  to  uniform  quantitative  measurement.  These 
quality  factors  remain  important  to  any  software 

but  one  of  a  number  of 
software  evaluation  tools. 

17-2 .  Metric  set 

minimum  set  of  12  metrics  is  described  in 
this  section.  Two  additional  metrics  (manpower  and 

P^°^^®®®)  ®*’®  described  in  Section  ll  as 
optional.  These  two  metrics  were  moved  out  of  the 
minimum  set  because  they  are  among  the  most  costly 
m®trics  in  terms  of  data  collection,  and  they  are^ 
collected  as  part  of  existing  program 
development  or  test  and  evaluation  activities. 

ll  ^be  metrics  fall  into  three  general  cateaopie« 

Issu^'^^'d®'  P''°9raiunatie  and  overall  management 
issues.  Requirements  metrics  pertain  to  the 
specification,  translation,  and  volatility  of 
requir®m®nts.  Quality  metrics  deal  with  testina  and 
other  software  technical  characteristic.  ® 

17-3.  Application 

KB  ^^®®®  metrics  apply  to  systems  governed  bv  the 

^70  series  of  regulations  and  those  governed  by  the 
AR  25  series  of  regulations.  ^ 

b.  Figure  17-1  shows  the  applicability  of  the 
minimum  set  of  metrics  over  the  life  cycle.  These 
®®^^^?®  provide  valuable  insight  into  a  orocn-ain 
especially  with  regard  to  demonstrated  results  ?Sd' 
readiness  for  test.  The  results  of  all  metrics  sLuld 
Program  Reviews,  In  Process  Reviews  and 

metrics  and  many  sub-elements  of  the  major  ones 
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17-1.  Matrics  and  Priauiry  Charaetaristies 


Metric 

Objective 

1  Managraant  Catagory  I 

Cost 

Track  s/w 
expenditures 

$  spent  vs 
$  allocated 

Schedule 

Track  schedule 
adherence 

Milestone /event 
slippage 

Computer  Resource 
Utilization 

Track  planned  and 
actual  resource 
use 

%  resource  capacity 
utilized 

Software  Engineering 
Environment 

Quantify  developer 
S/W  engineering 
environment 
maturity 

Computed  maturity 
level 

1  Requireaenta  Category  | 

Regts  Traceability 

Track  reqts  down 
to  code 

%  reqts  traced 

Reqts  Stability 

Track  changes  to 
reqts 

#  reqts  changes 

II  Quality  Category 

Design  Stability 

Track  design 
changes 

Stability  index 

Complexity 

Assess  code 
quality 

Complexity  indices 

Breadth  of  Testing 

Track  testing  of 
reqts 

%  reqts  tested, 

%  reqts  passed 

Depth  of  Testing 

Track  testing  of 
code 

Degree  of  code 
testing 

Fault  Profiles 

Track  open  vs 
closed  anomalies 

#  and  types  of 
faults,  average 
open  age 

Reliability 

Assess  S/W  mission 
failures 

Measure  down  time 

MTBF 

Restoration  times 

however,  are  not  suitable  for  presentation  at  high 
level  decision  reviews  such  as  ASARCs,  MAISRCs,  and 
Defense  Acquisition  Boards  (DABs) .  Figure  17-1  also 
identifies  the  metrics  which  should  be  reported  at  each 
major  decision  milestone.  Any  other  metric  which 
indicates  the  potential  for  serious  problems  should 
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Metrics  Purina  th^  Ufa  Cycla 


Aetlvltl88 


R«quir«m«nts| 

Rcqu  Anatynsl 

Coding.  Developer  I 

*  k 

Definition  | 

•nd  Design  | 

and  Govt  Teadng  | 

PDSS  1 

Mll8ttonM 

MSO 

MSCR 

/MS  MSI 

POR 

[AiSMSIl 

MS  III 

MS  1 

MSCR  MSI 

COR 

Mttrict 


Management  Metrics 

Cost 

Schedule 

Cornp  Resource  UHiz 
Si/W  Er>o  Environment 

Reqts  Metrics 
Reqts  Traoeablllty 
Reqts  Stabiity 

Quality  Metrics 
Design  Stability 
Compieidty 
Breadth  of  Testing 
Depth  of  Testing 
Fault  Profiles 
Reliability 


e  During  PDSS.  continue  to  apply  the  same  metrics  as 
were  applied  during  similar  actMtles  pre-M8  Hi 
e  Report  at  Milestone  Review 

Apply  If  prototype  software  wUI  be  used  In  later  phases 
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also  be  reported  at  this  time. 

17-4.  Other  eonsideratioBs 

a.  The  metric  descriptions  are  written  as  if  the 
software  is  being  developed  by  a  contractor* 

Substitute  "developer”  or  "developing  agency"  when 
software  is  procured  from  another  Army  or  Government 
agency. 

b.  The  applicable  time  periods  for  data  collection 
and  analysis  are  provided  with  each  metric  description. 
For  most  metrics,  data  should  continue  to  be  collected 
after  the  system  is  fielded.  Measure  software  updates 
produced  through  a  Life  Cycle  Software  Engineering 
Center  (LCSEC)  in  a  similar  fashion  as  software  in  a 
developmental  system.  As  a  general  rule,  collect  the 
same  data  in  PDSS  that  was  collected  in  similar 
activities  prior  to  Milestone  III. 

c.  The  metrics  should  be  used  as  program  maturity 
status  indicators.  They  should  be  used  to  portray 
trends  over  time,  rather  than  placing  too  much 
importance  on  a  calculated  value  at  a  single  point  in 
time.  Trends  can  be  studied  in  and  of  themselves,  or 
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*’•  '““Pore'S  with  trends  from  similar  systems 
that  have  already  been  built.  systems 

sunniioS^iJ®  of  thumb  for  evaluating  a  metric  are 

^  possible  or  sensible  to  do  so.  To 

date,  there  has  not  been  widespread  or  consistent  use 

?ir:°St?nn"er?t“'^^"  Spon°Shi“S"o  SHe 

criteria.  As  the  metrics  in  this 

nSTSilh^h^^hS^r®"'  '"'“'Sote'S  entities,  additional 

T^irSiJl  i!®);  *?  “ee*"^®  criteria. 

«  added  in  future  revisions  of  the 

*  validated  set,  however,  the 

indicators"*^"  useful  when  used  as  trend 

than’orif^JIf^-*^^^®  requirements  elements  are  used  by  more 
dSta  Ind  should  be  taken  that  consistent 

f  throughout  the  process. 

J  graphical  displays  shown  in  this  chapter  are 
Illustrative  purposes.  There  may  be  other 
ways  of  processing  and  displaying  the  data  that  are 
more  appropriate  for  a  specific  system. 

g.  Many  of  the  terms  describing  the  metrics  and 
5!}?^^  attributes  are  those  used  in  MSCR  systems.  Their 
AIS  equivalents  may  be  substituted  in  applying  the 
metrics  to  non-theater/tactical  information  systems. 

chapter,  the  terms  "CSU”  and 
module"  are  used  interchangeably. 

1.  To  facilitate  consistent,  unbiased,  and 

gathering  and  reporting,  a  survey  of 
commercially  available  software  tools  from  both 

industry  was  undertaken.  The  data 
requirements  in  this  chapter  that  are  captured  bv  each 

icntifirnd  in  the  survey.  The  results”! 
the  study  are  provided  as  Appendix  D. 

17-5.  Cost  metric 

•  ^^V^Pose/description.  This  metric  provides 

i!^clStron!d  ""irii  S'?®  °*  development 

IS  controlled*  it  is  directly  analooous  to  th© 

Cost/Schedule  Control  Systems  Criteria  (C/SCSC) 

i"  7000.2,  Performsnc.  Meisiremeit  for 
elected  Acquisitions,  already  in  use  to  track  the 

cost,  schedule,  and  technical  performance  of  hardware 
and  weapon  system  developments^  naraware 

b.  Life  cycle  application.  Begin  data  collection 
at  program  start  and  continue  through  fielding. 

Work  *  display .  MIL-HDBK-ras .  SW, 

work  Breakdown  Structure  for  Software  Elements. 

provides  guidance  in  developing  well  defined  work  and 
cost  accounting  packages  dealing  specifically  with 
software  effort  and  materials.  It  amplifies^the  " 
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become  a  cost  and  schedule  driver  for  many  systems. 
Development  and  test  of  software  is  highly  labor 
intensive.  Modern  management  practices  utilize  work 
breakdown  structures  to  help  provide  visibility  into 
and  manage  risk  involved  with  software  developments. 
This^metric  looks  at  the  effects  of  several  of  the  more 
volatile  software  work  categories  to  assess  overall 
progress  in  meeting  contract  cost  and  schedule  goals. 

(2)  Activity  types  describe  the  type  of  effort 
associated  with  the  collected  data.  Definitions  for 
rne  work  performed  in  each  activity  type  are  provided 
in  MIL-HDBK-WBS.SW,  s.cond  Draft,  datid  1  1I91. 

reporl2d“rd®te«JSd:*‘’“'”'^’’’  activity  types  should  be 

Requirements  analysis 
Design 

Code  and  unit  testing 
CSC(s)  integration  and  testing 
Formal  qualification  testing 
CSC!  integration  and  testing 
—  Software  problem/ change  resolution 
testing^^  redesign,  recoding,  CSC(s)  re-integration  and 

Software  engineering  management 
Software  quality  assurance 
Software  configuration  management 
Verification  and  validation 

HIL-HDBK-WBS.sSr^*  '***  squipment  in 

(m)  New  equipment  and  facilities 

(n)  Software  data 

(o)  Total  (The  sum  of  items  a-n  above  plus 
the  de5eloper?r^*  related  costs  tracked  or  incurred  by 

(3)  Figure  17-2  shows  a  sample  graph  of  the  life 
cycle  cost  metric.  The  terms  shown  on  the  graph  and 
other  cost  trend  algorithms  are  defined  as  follows: 

(«)  Budgeted  cost  of  work  scheduled  (BCWS)  - 
The  sum  of  the  budgets  for  all  work  packages,  the  level 
®^^®rt,  and  apportioned  effort  scheduled  to  be 
accomplished  within  a  given  time  period. 

Budgeted  cost  of  work  performed  (BCWP)  - 
The  sum  of  the  budgets  for  completed  work  packages  and 
completed  portions  of  open  work  packages,  plus  the 
applicable  portions  of  the  budgets  for  level  of  effort 
and  apportioned  effort. 

(c)  Actual  cost  of  work  performed  (ACWP)  - 
The  cost  actually  incurred  in  accomplishing  the  work 
performed  within  the  given  time  period. 

(^)  From  these  input  values,  performance  values 
and  trends  can  be  calculated: 

(a)  The  difference  between  planned  and  actual 

cost: 


(a) 

(b) 

(c) 

(d) 

(e) 

(f) 

(g) 


(h) 

(i) 

(j) 
(3c) 
(1) 


Cost  variance 


BCWP  -  ACWP 
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Cost 


Millions 


Program  Month 


-+-BCWS  -*-BCWP  -^ACWP 


Plgura  17-2.  8aapla  Cost  Matrie 


(b)  The  difference  between  the  amount  of  work 
planned  to  be  completed  and  actually  completed: 

Schedule  variance  «  BCWP  -  BCHS 

(5)  The  cost  and  schedule  variances  for  the 
project  of  Figure  17-2  are  portrayed  in  Figure  17-3. 

It  is  over  cost  and  behind  schedule.  Note:  Variances 
of  zero  would  mean  that  the  planned  budget  and  schedule 
were  being  met. 

d.  Data  requirements.  Note:  Use  cximulative  to 
date  values,  not  current  reporting  period  values  for 
all  cost  data  supplied. 

(1)  For  each  CSCI,  for  each  activity  type: 
(Collect  data  on  the  following  activities  at  the  CSCI 
level:  requirements  analysis,  design,  code  and  unit 
testing,  CSC(s)  integration  and  test,  FQT,  software 
problem/ change  resolution.) 

(a)  CSCI  identifier 

(b)  Activity  type  name 

(c)  BCWS 
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Cost  Performance  Tranris 


Millions 


Program  Month 
•+■  Schedule  Var  ^  Cost  Var 


Figrura  17-3.  Sanpla  Cost  Parfonaanca  Trands 

(d)  BCWP 

(e)  ACWP 

(2)  For  the  project/ system: 

(a)  For  each  activity  type:  (Collect  data  on 
the  following  activities  at  the  system  level:  CSCI 
integration  and  testing,  software  engineering 
management,  software  quality  assurance,  software 
configuration  management,  verification  and  validation, 
tools,  new  equipment  and  facilities,  software  data.) 

-  Activity  type  name 

-  BCWS 

-  BCWP 

-  ACWP 

(b)  Project  totals  of: 

-  BCWS 

-  BCWP 

-  ACWP 

e.  Frequency  of  reporting.  Monthly. 

f.  Use/ interpretation. 

(1)  Software  development  costs  include  costs 
from  all  of  the  activities  described  in  the  algorithm/ 
graphical  display  section  above. 

(2)  Software  engineering  management  includes  the 
planning  and  control  of  all  software  engineering 
efforts,  excluding  the  actual  design  and  hardware 
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extent  of  its  direct 

efforti?  software  engineering 

(3)  Software  quality  assurance  costs  include  all 
costs  associated  with  the  quality  support  teaS 
including -attendance  at  configuration  audits,  document 
reviews,  and  related  activities. 

an  nrsitl  validations  costs  include 

»  costs  include  the  software  reouired  to 

support  and  maintain  the  system  while  not  directly 
employed  in  the  operation  of  the  system  in  its  intended 
lib?a?ies^‘^^  compilers,  CASE  tools,  and  test  data^ 

(6)  Equipment  and  facilities  include  the  entire 

and  test  p?o"an 
Support  software  development  or 
maintenance  including  microprocessor  development 

perforaance  analyzers,  diagnostic  equipment, 
and  modernization  of  facilities.  ^  ^ 

Software  data  costs  are  those  associated 

technical  data  requirements.  This  activity  type  does 

analisii^fnr^^^®H®^^°^^®  design,  requirements 

software  coding,  required  in  developing  the 

(8)  This  metric  is  used  to  track  software 
expenditures  versus  allocations  over  the  life  of  a 
program.  Status  needs  to  be  determined  not  only  on  the 
present  percent  of  allocation  used,  but  also  in 
relation  to  what  has  been  done  to  date  and  how  much  is 

deLh°of®<-‘^°?®  breadth  of  testing, 

epth  of  testing,  reliability,  manpower  f optional ^  anH 

?fsk  cfrs*  (optional)).  Furthir  insight 'into 

J  determined  by  examining  expenditures 
relating  to  re-work  (i.e.,  fixing  faults  and 
requirements  changes) .  Exceeding  the  allocation  at  anv 
point  in  time  is  cause  for  concern  and  investigation. 

data  should  also  be  interpreted  in  the 
^*'®  events  and  deliveries  scheduled 

^5®  Undesirable  cost  trends  may  signal 

impending  delays  of  major  events  or  milestones.  A 
sample  program  schedule  is  shown  as  Figure  17-4. 

(10)  Trends  in  cost  and  schedule  variances 
should  be  tracked  over  time.  Consistently  or 
increpingly  negative  values  for  the  variances  are 
signals  that  the  project  may  be  delivered  behLd 

variance)  or  exceed  budget 
at  its  originally  scheduled 

time  of  completion. 

g.  Rules  of  thumb.  No  formal  threshold  values  are 
given.  Management  attention  needs  to  be  heightened 
whenever  expenditures  are  nearing  allocated  values  and 
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Figure  17-4.  Sample  Program  schedule 

much  work  is  yet  to  be  done.  A  program  review  may  be 
necessary  in  this  case  and  should  be  mandatory  when  an 
allocation  is  actually  exceeded. 

h.  References.  References  (a)  and  (j)  of  Section 
III  are  applicable  to  this  metric. 

17-6.  Schedule  metric 

a.  Purpose/description.  The  schedule  metric 
indicates  changes  and  adherence  to  the  planned 
schedules  for  major  milestones,  activities  and  key 
software  deliverables. 

b.  Life  cycle  application.  Begin  collecting  data 
at  program  start,  and  continue  for  the  entire  software 
development. 

c.  Algorithm/graphical  display. 

(1)  Plot  planned  and  actual  schedules  for  major 
milestones  and  key  software  deliverables  as  they  change 
over  time. 

(2)  On  the  sample  graph  shown  in  Figure  17-5, 
the  PDR  and  COR  milestone  schedules  are  plotted  over 
time. 

Any  milestone  or  event  of  interest  can  be  plotted. 
Similar  plots  can  also  be  made  for  key  product 
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Schedule 


Planned  Date 


Program  Month 


Figure  17-5.  Sanpla  Schadula  Natric 

deliverables  (e.g.,  Software  Product  Specification 
(SPS)).  To  read  the  graph,  find  the  actual  date 
(program  month)  on  the  x-axis,  and  read  the  appropriate 
planned  date  on  the  y-axis.  For  example,  at  month  one, 
the  POR  was  planned  for  month  two,  and  the  CDR  was 
planned  for  month  eight.  At  month  two,  the  PDR 
schedule  has  slipped  to  month  three  (a  slip  of  one 
month) ,  whereas  the  CDR  schedule  has  remained  the  same. 
At  month  three,  the  PDR  schedule  has  slipped  to  month 
five  (an  additional  slip  of  two  months),  whereas  the 
CDR  schedule  has  remained  the  same. 

(3)  Longer  duration  events  can  be  plotted  as 
shown  in  Figure  17-6  to  reflect  changes  in  event 
duration  as  well  as  changes  in  expected  starting  dates. 
An  example  would  be  tracking  the  various  software 
development  phases. 

(4)  A  table  showing  the  schedule  and  status  of 
starting  and  ending  dates  for  key  activities  and  events 
can  be  produced  as  illustrated  in  Table  17-2.  A 
negative  entry  in  a  "slip"  column  indicates  that  the 
date  has  been  moved  earlier  in  time. 

d.  Data  requirements. 

(1)  Major  milestone  schedules.  Examples  of 
milestones  are  SDR,  SRR,  SSR,  PDR,  CDR,  FQT,  FCA,  PCA, 
ASARC,  and  MAI SRC. 
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(2)  Delivery  schedule  for  key  software 
deliverables.  Examples  of  software  deliverables  are 
SDP,  SPS,  SSS,  IRS,  VDD,  SRS,  SDD,  STP,  STD,  and  STR. 

(3)  Software/system  program  schedules.  Examples 
of  other  software  or  system  activities  and  events  of 
interest  are  CSCI  development  (or  its  subcomponents, 
requirements  analysis,  design,  code  and  unit  test, 
etc.),  CSCI  integration  test,  DT  and  OT. 

(4)  For  the  reporting  period:  For  each 
milestone,  deliverable,  event  or  activity  to  be 
tracked: 

(a)  Event  name 

(b)  Date  of  report 

(b)  Date  the  event  is  planned  to  start 

(c)  Date  the  event  is  planned  to  end 

(d)  Date  the  event  actually  started 

(e)  Date  the  event  actually  completed 

e.  Frequency  of  reporting.  Monthly.  -/e, 

f.  Use/ interpretation.  -^n  over 

(1)  The  schedule  metrics,  when  plotted 
change  over  time,  provide  indications  of  problems  'in 
meeting  key  events  or  deliveries.  The  higher  the  slope 

line  for  each  event,  the  more  problems  are 
being  encountered.  Milestone  slippages  should  be 
investigated.  Potential  clustering  (i.e.,  bunching-up 
in  time)  of  key  events  should  be  guarded  against. 

(2)  The  schedule  metric  can  be  used  in 
conjunction  with  several  other  metrics  to  help  judge 
program  risk.  For  example,  it  could  be  used  with  the 
test  coverage  metrics  to  determine  if  there  is  enough 
time  remaining  on  the  current  schedule  to  allow  for  the 
completion  of  all  testing. 

The  schedule  metric  passes  no  judgement  on 
the  achievability  of  the  contractor’s  development  plan. 

g.  Rules  of  thumb.  No  formal  evaluation  criteria 
^OT  the  trend  of  the  schedule  metrics  are  given. 

However,  large  slippages  are  indicative  of  problems. 
Maintaining  conformance  with  calendar-driven  schedules 
should  not  be  used  as  the  basis  for  proceeding  beyond 
milestones.  ^ 

h.  References.  None. 

I?-?.  Computer  resource  utilisation  metric 

a.  Purpose/description.  This  metric  is  intended  to 
show  the  degree  to  which  estimates  and  measurements  of 
the  target  computer  resources  (central  processing  unit 
(CPU)  capacity,  memory/ storage  capacity,  and 
input/output  (I/O)  capacity  are  changing  or  approaching 
the  limits  of  resource  availability  and  specified 
constraints.  Over-utilization  of  computer  resources 
can  have  serious  impacts  on  cost,  schedule,  and 
supportability.  Approaching  resource  capacity  may 
necessitate  hardware  change  or  software  redesign. 
Exceeding  specified  reserve  requirements  can  have 
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similar  impacts  in  the  post-deployment  phase.  Proper 
use  of  this  metric  can  also  assure  that  each  processor 
in  the  system  has  adequate  reserve  to  allow  for  future 
growth  due  to  changing  or  additional  requirements 
without  requiring  re-design.  This  metric  can  be 
applied  to  a  system  architecture  which  is  distributed 
or  centralized. 

b.  Life  cycle  application.  Early  in  the  design 
phase,  CPU  utilization  budgets  should  be  established 
for  each  processor  in  the  system.  I/O  utilization  and 
throughput  budgets  should  be  allocated  to  each  I/O 
channel  in  the  system.  Memory/ storage  usage  budgets 
should  be  allocated  to  all  CSUs,  CSCs,  CSCIs  and 
temporary  and  permanent  data  files  early  in  the  design 
phase.  Based  on  these  estimates  CPU,  memory,  and  I/O 
allocations  must  be  documented  in  the  Software 
Requirements  Specification  prior  to  the  Software 
Specification  Review,  and  must  be  documented  in  the 
Software  Design  Document  prior  to  the  Critical  Design 
Review.  All  changes  to  these  initial  estimates  should 
be  reported,  including  those  caused  by  hardware 
modifications.  For  the  memory  and  I/O  categories, 
actual  usage  should  be  measured  monthly  during  coding, 
unit  testing,  integration  testing,  CSCI  testing  and 
system-level  testing.  For  the  CPU  usage  statistic, 
measurements  should  be  taken  monthly  after  the 
beginning  of  unit  testing.  Actual  utilization  should 
be  formally  demonstrated  at  the  system  level  for  each 
resource  under  peak  loading  conditions  during  FQT,  and 
during  PDSS  if  additional  capability  is  added. 

c.  Algorithm/graphical  display. 

(1)  The  allocation  for  each  resource  type  should 
not  exceed  the  target  upper  bound  utilization  for  any 
category  (for  some  systems,  the  allocation  equals  the 
target  upper  bound) . 

(2)  CPU  and  I/O  resource  utilization  are 
typically  measured  by  the  system.  While  the 
measurements  contribute  slightly  to  system  overhead, 
the  feature  typically  comes  with  the  system  it  its  off 
the  shelf  configuration.  In  instances  where  the  system 
does  not  measure  itself  in  terms  of  CPU  and  I/O 
utilization,  the  percent  utilization  must  be  computed. 

(3)  For  memory/storage  resources,  the  percent 
utilization  must  be  computed.  For  memory,  the  resource 
is  random  access  memory  (RAM) ,  RAM  for  this  metric 
refers  to  both  volatile  and  non-volatile  (including 
read  only)  memories.  For  storage,  the  resources 
include  disk  space  and  other  mass  storage. 

(4)  As  software  development  proceeds,  the 
measured  values  for  each  category  should  be  projected 
out  to  the  ''full"  system.  For  example,  if  half  of  the 
"size"  of  the  software  is  built  and  measured,  the 
projected  value  for  utilization  would  be  the  actual  for 
the  portion  built  and  measured  to  date,  plus  the 
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budgeted  portion  yet  to  be  added. 

(5)  In  the  sample  graph  of  Figure  17-7,  target 
upper  bound  utilization  is  shown  as  a  straight  line, 
in  reality,  the  target  upper  bound  utilization  can 
time.  The  sample  shown  represents  the 
®  single  CPU  resource;  similar  graphs 
should  be  constructed  for  the  utilization  of  all  other 
CPUs  plus  each  I/O  and  memory  resource. 


Computer  Resource  Utilization 


Figure  17-7.  Sample  Computer  Resource  Utilization 


d.  Data  requirements.  Actual  usage  should  be 
measured  during  pea)c  operational  loading  periods  and 
should  include  the  operating  system  and  non-developer 
supplied  software  as  well  as  the  development  software. 
Similarly,  projected  usage  should  be  based  on  estimates 
using  peak  operational  loading  periods  and  should 
include  the  operating  system,  development  software  and 
non-developer  supplied  software.  Where  ’’target"  is 
used  in  these  data  elements,  it  is  actually  meant  to 

include*  upper  bound.  The  data  requirements 

(1)  For  each  CPU  in  the  system: 

(a)  CPU  identifier 

(b)  Target  CPU  usage  (%  of  capacity) 

(c)  Actual  CPU  usage  (%  of  capacity) 

(d)  Projected  CPU  usage  (%  of  capacity) 
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(2)  For  each  I/O  channel  in  the  system: 

(a)  I/O  channel  identifier 

(b)  Target  I/O  usage  (%  of  capacity) 

(c)  Actual  I/O  usage  (%  of  capacity) 

(d)  Projected  I/O  usage  (%  of  capacity) 

(3) .  For  each  RAM  in  the  system  (in  bytes): 

(a)  RAM  identifier 

(b)  Capacity 

(c)  Target  upper  bound 

(d)  Projected  usage 

(e)  Actual  usage 

(4)  For  each  mass  storage  device  in  the  system 
(in  bytes):  . 

(a)  Storage  device  identifier 

(b)  Capacity 

(c)  Target  upper  bound 

(d)  Projected  usage 

(e)  Actual  usage 

(5)  For  each  CSCI  in  the  system: 

(a)  CSCI  identifier 

(b)  RAM  allocation  budget  as  specified  in  the 
requirements  (in  bytes) 

(c)  Actual  RAM  usage  (in  bytes) 

(d)  Mass  storage  allocation  budget  (in  bytes) 

(e)  Actual  mass  storage  usage  (in  bytes) 

e.  Frequency  of  reporting. 

(1)  CPU,  I/O,  memory/ storage  allocation  — 
monthly  starting  with  SSR. 

(2)  CPU,  I/O,  memory/ storage  usage  (actual, 
projected,  and  target  upper  bound)  —  monthly  starting 
with  CDR. 

f.  Use/ interpretation. 

(1)  Resource  utilization  tends  to  increase  over 
the  development  of  a  project.  Therefore,  adequate 
planning  must  be  done  to  ensure  that  the  software's 
operation  does  not  put  undue  demands  on  the  target 
hardware !s  capabilities.  This  measure  tracJcs 
utilization  over  time  to  ma)ce  sure  that  target  upper 
bound  utilization  is  not  exceeded  and  that  sufficient 
excess  capacity  remains  for  future  growth  and  for 
periods  of  high  stress  loading. 

(2)  In  multi-processor  environments,  each 
processor  should  be  trac)ced  separately,  and  each  should 
be  allocated  a  planned  utilization. 

(3)  In  instances  where  the  development  and 
target  environments  differ  in  types  and/or  capacities, 
caution  must  be  ta)cen  in  computing  and  analyzing  the 
®'®^sures.  Translations  are  acceptable  up  to  a  certain 
point,  but  testing  on  the  target  hardware  must  taJce 
place  as  early  as  possible. 

(4)  Tailoring  may  be  appropriate  for  situations 
when  dynamic  allocation,  virtual  memory,  parallel 
processing,  multitasicing,  or  multiuser-based  features 
are  employed. 
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(5)  Initial  estimates  should  be  retained  for 
comparison  with  what  is  finally  achieved  in  order  to 
aid  in  scoping  future  programs. 

(6)  In  addition  to  collecting  utilization  data 
throughout  the  build-up  of  contractor  testing 
(including  single  thread  to  multiple  thread), 
measurements  should  also  be  taken  during  system  level 
stress  testing. 

(7)  During  development,  it  is  important  to  look 
at  both  actual  and  projected  values  in  relation  to  the 
target  upper  bound  values.  If  either  exceeds  the 
target  values,  e^ra  attention  should  be  paid  to  assure 
that  the  projections  fall  below  the  target  upper  bound 
value  by  project  completion.  If  it  is  apparent  from 
the  projections  that  the  target  upper  bound  limits  will 
be  exceeded,  action  must  be  taken  to  either  optimize 
the  software  or  upgrade  the  capability  of  the  target 
configuration. 

(8)  Sudden  drops  in  utilization  may  reflect 
either  new  systems  capacity  or  new  software  that 
embodies  more  efficient  programming. 

(9)  Computer  resource  utilization  metrics  should 
be  used  in  conjunction  with  the  test  coverage /success 
metrics  (breadth  and  depth  of  testing)  to  ensure  that 
measures  of  the  actual  usage  are  representative  and 
portray  the  entire  system  under  realistic  stress  loads. 
If  the  development  progress  metric  is  available, 
computer  resource  utilization  metrics  should  be  used  in 
conjunction  with  development  progress  to  ensure 
remaining  development  can  be  accomplished  without 
exceeding  planned  utilization. 

(10)  Computer  resource  utilization  provides  the 
link  to  total  system  performance.  As  mentioned 
previously,  on  many  computer  platforms,  the  data  are 
relatively  easy  to  collect  (indeed,  often  self- 
measured)  ,  and  are  often  built  in  to  the  overhead  of 
the  system.  On  other  platforms  this  may  not  be  the 
case.  Collecting  computer  resource  utilization  data 
will  contribute  some  small  amount  to  system  overhead. 

g.  Rules  of  thumb. 

(1)  Rules  of  thumb  depend  on  the  technology 
employed  for  each  system.  When  using  processors 
designed  to  operate  at  100%  of  capacity,  design  for 
90%,  to  allow  for  the  increase  in  planned  utilization 
which  almost  inevitably  occurs  during  design  and 
development.  Actual  utilization  can  approach  or, 
depending  on  the  processor,  equal  100%  with  no 
deterioration  in  performance. 

(2)  For  processors  which  are  not  designed  to  run 
efficiently  at  100%  of  capacity,  one  must  be  careful 
not  to  approach  100%  utilization.  For 

embedded/ tactical  systems,  design  for  no  more  than  50% 
utilization  for  memory/ storage,  CPU,  and  I/O  resources. 
For  information  area  systems,  a  higher  target  upper 
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bound  value  may  be  allowable,  but  should  be  specified 
in  the  requirements  documents.  Performance  may 
deteriorate  when  utilization  exceeds  70%  for  time- 
applications.  Schedule  and  cost  can  be 
severely  impacted  as  utilization  exceeds  90%. 

(3).  For  systems  employing  virtual  memory 
architectures,  usage  of  RAM  is  not  as  important  as 
measuring  the  amount  of  swapping  that  occurs  during 
peak  load  periods.  ^ 


(4)  If,  at  any  time  during  development,  actual 
or  projected  computer  resource  utilization  exceeds 
target  upper  bound  utilization,  an  immediate  review 
must  be  held.  Corrective  action  (e.g.,  software  re¬ 
design,  hardware  upgrades,  etc.)  must  be  taken  before 
proceeding  to  the  next  stage  of  development. 

h.  References.  References  (a)  and  (g)  of  Section 
m  are  applicable  to  this  metric. 


3-7-8.  Software  eagiaeerlag  eaviroameat  metric 

a.  Purpose/description.  The  software  engineering 
environment  metric  provides  a  rating  of  the 
contractor's  applied  software  engineering  principles. 
Examples  of  these  principles  are  the  use  of  structured 
design  techniques,  the  extent  of  tool  usage,  and  the 
use  of  program  design  language  (PDL) .  If  practical, 
aspects  of  the  methodology  could  also  be  applied  to 
®ateriel  developer  personnel  or  the  program  manager ' s 
®atrix  support  staff  for  the  purpose  of  assessing 
capabilities  with  respect  to  software  development. 
Rating  of  software  developers  should  be  performed  by  a 
qualified  independent  group.  Performing  a  software 
engineering  environment  assessment  is  described  in 
report  CMU/SEI-87-TR-23.  The  PM  is  responsible  for 
ensuring  that  the  proper  methodology  is  used  during  the 
assessment  of  software  process  maturity. 

b.  Life  cycle  application.  At  each  point  a 
software  contractor  is  brought  on  to  a  program. 

Reassess  any  previously  identified  risks  at  major 
milestones  to  monitor  improvements  in  the  contractor's 
software  engineering  process  maturity. 

c.  Algorithm/graphical  display.  Follow  the 
methodology  outlined  in  CMU/SEI-87-TR-23,  which 
includes  the  following: 

(1)  Collect  questionnaire  data  from  contractor. 

p)  Conduct  follow-up  visit  (by  the  assessment 
team)  to  answer  further  questions,  observe  tools,  etc. 

(3)  Perfomn  assessment. 

(4)  Calculate  the  process  maturity  level.  The 
levels  are  broadly  defined  as  possessing  the  following 
characteristics: 

(a)  Initial  (level  one) 

-  ill-defined  procedures  and  controls 

—  no  consistent  application  of  software 
engineering  management  to  the  process 
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-  no  use  of  modern  tools  and  techniques 

(b)  Repeatable  (level  two) 

-  management  of  costs  and  schedules 

-  use  of  standard  methods  and  practices 
for  managing  some  software  development 
activities 

(c)  Defined  (level  three) 

-  software  development  process  is  well- 
defined  in  terms  of  software  engineering 
standards  and  methods 

-  increased  organizational  focus  on 
software  engineering 

-  use  of  design  and  code  reviews 

-  internal  training  programs 

-  establishment  of  software  engineering 
process  group 

(d)  Managed  (level  four) 

-  software  development  process  is 
quantified,  measured,  and  well- 
controlled 

-  decisions  are  based  on  quantitative 
process  data 

-  use  of  tools  to  manage  design  process 

-  use  of  tools  to  support  data  collection 
and  analysis 

-  accurate  projection  of  expected  errors 

(e)  Optimized  (level  five) 

-  major  focus  on  improving  and  optimizing 
process 

-  sophisticated  analysis  of  error  and  cost 
data 

-  employment  of  error  cause  analysis  and 
prevention  studies 

-  iterative  improvement  of  process 

d.  Data  requirements.  Responses  to  questionnaire 
and  rating  results. 

e.  Frequency  of  reporting.  As  needed  to  evaluate 
software  process  maturity  standing. 

f.  Use/ interpretation. 

(1)  The  software  engineering  environment  rating 
provides  a  consistent  measure  of  the  capability  of  a 
contractor  to  use  modern  software  engineering 
techniques  in  his  development  process,  and  therefore 
his  capability  to  instill  such  principles  and 
characteristics  in  the  product.  The  basic  assumption 
to  this  approach  is  that  a  quality  process  results  in  a 
quality  product.  Other  metrics,  as  well  as  other 
evaluation  techniques,  should  be  used  to  examine  the 
quality  of  the  product. 

(2)  The  software  engineering  environment  rating 
will  be  used  to  validate  the  correlation  between 
software  process  maturity  and  the  quality  of  the 
resulting  software  product. 
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(3)  The  use  of  the  software  engineering 
environment  rating  may  encourage  contractors  to  improve 
weaknesses  in  their  software  development  process  in 
order  to  increase  their  rating.  A  higher  rating  will 
increase  the  contractor's  chance  of  being  selected  for 
future  software  development  projects. 

g.  Rules  of  thumb.  None.  The  metric  will  assist 
in  validation  of  the  Army  metric  program  only. 

However,  the  Software  Engineering  Institute  reports 
that  currently,  only  a  very  small  percentage  (i.e.,  2- 
3%)  of  companies  have  achieved  ratings  of  level  3,  4, 
or  5.  Most  companies  are  rated  at  level  l  or  2.  A 
value  of  1  is  the  lowest  maturity  level  and  5 
represents  the  highest  level. 

h.  References.  Reference  (e)  of  Section  III  is 
applicable  to  this  metric. 

17-9.  Requirements  traceability  metric 

a.  Purpose/description.  The'  requirements 
traceability  metric  measures  the  adherence  of  the 

products  (including  design  and  code)  to  their 
requirements  at  various  levels.  It  also  aids  the 
combat  developer,  materiel  developer,  and  evaluators  in 
determining  the  operational  impact  of  software 
problems . 

b.  Life  cycle  application.  Begin  tracing  during 
user  requirements  definition  phase.  Update  the  trace 
in  support  of  major  milestones,  such  as  PDR  and  CDR,  or 
at  major  software  release  points. 

c.  Algorithm/graphical  display. 

(1)  This  metric  is  a  series  of  percentages, 
which  can  be  calculated  from  the  matrix  described 
below.  Note:  The  term  software  requirements  as  used 
in  this  metric  includes  any  software  interface 
requirements. 

(2)  Trace  all  ORD  requirements  to  the  UFD. 
Identify  any  ORD  requirements  not  found  in  the  UFD. 
Calculate  the  percentage  of  ORD  requirements  in  the 
UFD . 

(3)  Trace  all  lowest  level  UFD  requirements  to 
the  system  level  specification,  and  trace  all  software- 
related  UFD  requirements  to  the  software 
specifications.  Identify  any  omissions.  Calculate  the 
percentage  of  software  UFD  requirements  that  are  in  the 
software  specification. 

(4)  The  software  requirements  as  specified  in 
the  software  specifications  must  then  be  traced  into 
the  software  design,  code,  and  test  cases. 

(5)  The  technique  to  perform  this  analysis  is 
the  development  of  a  software  requirements  traceability 
matrix  (SRTM) .  The  SRTM  is  the  product  of  a 
structured,  top-down  hierarchical  analysis  that  traces 
the  software  requirements  through  the  design  to  the 
code  and  test  documentation.  The  SRTM  should  contain 
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enough  infonoation  so  that  the  relationship  between 
various  levels  of  requirements  and  the  requirements  to 
their  design  and  test  cases  can  be  observed.  A 
conceptual  example  of  a  SRTM  is  shown  in  Table  17-3. 
The  software  specifications  and  requirements  should  be 
listed  in-groups  which  represent  higher-order  system 
requirements,  in  this  manner,  the  grouping  of  CSUs 
which  represent  a  required  system  function  can  be 
readily  seen.  Also,  it  is  good  practice  to  trace  the 
requirements  at  additional  levels  between  design  and 
code.  For  example,  they  can  be  traced  to  functional 
decomposition  documents,  flow  diagrams,  data 
dictionaries,  etc.  (Note:  In  Table  17-3,  question 
marks  indicate  that  tracing  has  not  yet  occurred.  At 
each  level  of  the  trace,  a  single  requirement  can  be 
traced  to  multiple  lower  level  requirements.) 


Table  17-3.  Sample  Software  Requirements 
Traceability  Matrix 


ORO 

Reqt 

UFD 

Reqt 

sss 

Reqt 

SRS/IRS 

Rtqt 

Ofts  i  gn 

iMt 

Cast 

CSCI 

CSC 

CSU 

CQd« 

r1 

r1 

r1 

SRSI/rl 

CSCI1 

CSCI 

csui 

CSU1 

TCI 

r1 

r1 

r1 

SRSVrl 

CSCI1 

CSCI 

CSU4 

C$U4 

7 

r1 

rZ 

n 

SM1/r2 

CSCI1 

CSC2 

CSUS 

CSUS 

TC2 

n 

rZ 

r2 

SRS2/P1 

CSC12 

? 

? 

7 

? 

n 

rZ 

rZ 

IRSI/rl 

CSCI2 

? 

? 

7 

? 

r2 

• 

• 

• 

. 

. 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

(6)  The  degree  of  completion  of  the  SRTM  depends 
upon  the  current  stage  of  the  software  life  cycle  as  to 
which  documents  are  available.  The  SRTM  should  be  part 
of  the  technical  data  package.  From  the  SRTM,  various 
statistics  can  be  calculated  indicating  percentage  of 
tracing  to  various  levels.  Examples  of  such 
calculations  are: 

-  %  ORD  requirements  in  UFD 

-  %  software  UFD  requirements  in  SRS/IRS 

-  %  software  requirements  in  CSU  design 

-  %  software  requirements  in  CSC  design 

-  %  software  requirements  in  CSCI  design 

-  %  software  requirements  in  code 

-  %  software  requirements  having  test  cases 
identified 
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worthwhile  to  perform  a 

backwards  trace  (e.g.,  from  code  to  requirements).  In 

list  of  matrix,  one  can  simply  make  a 

list  Of  the  distinct  entries  and  compare  with  a  total 
count  of  the  entries  for  the  column  bf  interest.  To 
carry  the-example  of  doing  a  backwards  trace  from  code 
5k  further,  one  would  make  a  list  of  all 

555  appear  in  the  "code"  column  of 

555  list  should  then  be  compared  with  the 

555  5  system.  Any  CSU  which  does 

not  appear  in  the  "code"  column  may  not  support  any 
requirement.  These  CSUs  should  be  investigated. 

(8)  Baclcwards  tracing  of  requirements  at  the 
early  stages,  such  as  8RS  to  UFD,  can  identify 
requirements  that  have  been  added  in  the  requirements 
decomposition  process.  Each  requirement  in  a 
specification  should  be  attributable  in  some  way  to  a 

higher  level  requirement  from  a  predecessor 
requirement.  -  : 

d.  Data  recjuirements. 

(1)  Documented  requirements/specifications  (ORD. 

SRS,  IRS).  ' 

Documented  design  of  CSCIs,  CSCs,  CSUs  (SDD, 


UFD, 


IDD, 


SSS, 
(2) 
SPS) 
(3) 


D0D-STD-2167A^^'^**^*  description  in  accordance  with 

(4)  Completed  SRTM. 

(5)  Number  of  ORD  requirements: 

-  total 

-  traceable  to  UFD 

-  not  traceable  to  UFD 

(6)  Number  of  UFD  requirements: 

-  total 

*■  traceable  to  SSS 

-  not  traceable  to  SSS 

-  traceable  to  ORD 

-  not  traceable  to  ORD 

(7)  Number  of  UFD  software  requirements: 

-  total 

-  traceable  to  SRS/ IRS 

-  not  traceable  to  SRS/ IRS 

(8)  Number  of  SSS  software  requirements: 

-  total 

-  traceable  to  SRS/IRS 

-  not  traceable  to  SRS/IRS 
For  each  CSCI,  Number  of  SRS/IRS 


(9) 

requirements: 

total 


traceable  to  CSCI  design 
traceable  to  CSC  design 
traceable  to  CSU  design 
traceable  to  code 
covered  by  one  or  more  test  case 
not  traceable  to  UFD 
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®*  Frequency  of  reporting.  Update  periodically  in 
support  of  milestones  or  major  releases, 
f.  Use/ interpretation. 

(1)  As  can  be  seen  above,  the  tracing  of 
requirements  must  occur  at  several  levels.  This 
tracing  should  be  a  key  Government  tool  at  all  system 
requirement  and  design  reviews.  It  can  serve  to 
indicate  those  areas  of  requirements  or  software  design 
that  have  not  been  sufficiently  thought  out.  The  trend 
of  the  SRTM  should  be  monitored  over  time  for  closure. 

(2)  By  the  nature  of  the  software  development 
process,  especially  in  conjunction  with  an  evolutionary 
development  strategy,  the  trace  of  recpiirements  is  an 
^^®rative  process.  That  is,  as  new  software  releases 
add  more  functionality  to  the  system,  the  requirements 
trace  will  have  to  be  revisited  and  augmented. 

(3)  Although  not  portrayed  graphically  above, 
trends  of  requirements  traceability  can  be  shov- 
time.  For  example,  during  the  requirements  pha. 
percent  of  UFD  requirements  traced  into  the  SRS  can  be 
depicted. 

(4)  The  SRTM  is  heavily  tied  to  the  UFD,  a 
concept  recently  embraced  by  the  user  representative 
community.  The  regulatory  guidance  requiring  a  UFD  has 
not  yet  been  staffed  throughout  the  Army.  HoweveT, 
even  if  a  UFD  is  not  used,  the  SRTM  can  still  be 
created  and  used. 

(5)  One  of  the  important  new  characteristics 
that  will  be  embodied  in  the  UFD  is  grouping 
requirements  by  user  priority  levels.  Such  a 
prioritization  could  be  used  with  the  SRTM  to  highlight 
certain  key  user  functions. 

(6)  Another  benefit  of  requirements  traceability 
IS  that  those  modules  which  appear  most  often  in  the 
matrix  (thus  representing  the  ones  that  are  most 
crucial  in  the  respect  that  they  are  required  for 
multiple  functions  or  requirements)  can  be  highlighted 
for  earlier  development  and  increased  test  scrutiny. 

(7)  The  requirements  traceability  metrics  should 
be  used  in  conjunction  with  the  test  coverage  metrics 
(depth  and  breadth  of  testing)  and  the  development 
progress  metric  (optional)  to  verify  if  sufficient 
functionality  has  been  demonstrated  to  warrant 
proceeding  to  the  next  stage  of  development  or  testing. 
They  should  also  be  used  in  conjunction  with  the  design 
stability  and  requirements  stability  metrics. 

(8)  Due  to  the  detailed  nature  of  requirements 
traceability  metrics,  they  should  be  a  normal  product 
of  the  V&V  effort.  The  SRTM  may  be  developed  by  the 
contractor,  but  must  be  verified  by  an  independent 
agent  such  as  the  IViV  contractor. 

(9)  During  PDSS,  if  a  function  is  modified,  the 
SRTM  can  be  used  to  focus  regression  testing  on  a 
particular  CSCI/CSC/CSU. 
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g.  Rules  of  thumb. 

(1)  Do  not  approve  the  UFD  until  all  appropriate 
ORD  requirements  have  been  traced  into  it. 

(2)  Do  not  proceed  beyond  SSR  until  all 
software-related  UFD  requirements  have  been  traced  into 
the  software  requirements. 

(3)  Do  not  proceed  beyond  CDR  until  a  high  trace 
percentage  exists  from  the  ORD,  through  the  UFD, 
through  the  system  specification,  through  the  software 
specification,  to  the  design  at  CSCI,  CSC,  and  CSU 
level  has  been  achieved. 

(4)  Do  not  proceed  to  formal  Government  testing 
until  a  100%  trace  to  the  code  has  been  achieved. 

(5)  Do  not  start  formal  Government  testing 
unless  100%  of  the  SRS  requirements  are  traceable  to 
test  cases.  This  reflects  good  software  engineering 
practice  and  will  insure  that  the  software  deyeloper  is 
ready  to  start  formal  Government  testing. 

h.  References.  Reference  (f)  of  Section  III  is 
applicable  to  this  metric. 

17-10.  Requirements  stability  metric 

a.  Purpose/description.  The  metrics  on 
requirements  stability  indicate  the  degree  to  which 
changes  in  the  software  requirements  or  changes  in  the 
contractor's  understanding  of  the  requirements  affect 
the  development  effort.  It  also  allows  for  determining 
the  cause  of  requirements  changes. 

b.  Life  cycle  application.  Begin  collecting  during 
user  requirements  definition  phase.  Measure 
requirements  with  respect  to  the  UFD  between  MSCR 
Milestone  I  and  II  (user  responsibility) .  Measure 
requirements  with  respect  to  the  SRS  after  MSCR 
Milestone  II  (materiel,  developer  responsibility) . 

c.  Algorithm/graphical  display. 

(1)  Samples  of  the  requirements  stability  metric 
are  shown  in  Figures  17-8  and  17-9. 

(2)  The  first  chart  shows  cumulative 
requirements  discrepancies  over  time,  versus  closure  of 
those  discrepancies.  The  second  chart  is  a 
representation  of  the  effect  of  requirements  changes  on 
the  code  (percent  of  source  lines  of  code  (SLOC) 
changed  by  month) .  In  actuality,  one  should  develop 
several  versions  of  the  second  chart.  One  version 
should  show  the  n\imber  of  Engineering  Change 
Proposals  -  Software  (ECPs-S)  due  to  user  changes  and 
the  number  of  ECPs-S  due  to  developer  changes.  The 
second  version  should  show  the  percent  SLOC  affected  by 
developer  changes.  Additionally,  one  might  loo)c  at  the 
number  of  modules  affected  by  both  types  of 
requirements  changes. 

d.  Data  requirements.  The  difficulties  in  agreeing 
on  a  standard  definition  of  a  line  of  code  are  well 
recognized.  A  "line  of  code"  will  differ  from  language 
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Beouirements  Stability 
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Requirements  StahijjtY 

Percent  SLOCe  Chengcc 
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to  language,  as  well  as  from  programming  style  to 
programming  style.  In  order  to  consistently  measure 
source  lines  of  code  count  all  non-comment,  non-blank, 
executable  and  data  statements.  The  data  requirements 
are;  For  each  CSCI; 

(1) -  Number  of  software  requirements 
discrepancies  as  a  result  of  each  review  (SSR,  PDR, 

(2)  Monthly  status  (cumulative  total/total 
number  resolved)  of  above  discrepancies. 

(3)  Number  of  ECPs-S  generated  from  requirements 
changes  due  to  user. 

(4)  Number  of  SLOC  affected  by  approved  ECPs-S 
due  to  user  changes. 

(5)  Number  of  ECPs-S  generated  from  requirements 
changes  due  to  developer. 

(6)  Number  of  SLOC  affected  by  approved  ECPs-S 
due  to  developer  changes. 

(7)  Number  of  modules  affected  by  requirements 
changes  due  to  user. 

(8)  Number  of  modules  affected  by  requirements 
changes  due  to  developer. 

(9)  Total  niimber  of  SLOCs. 

(10)  Total  number  of  SRS  requirements. 

(11)  Number  of  SRS  requirements  added  due  to 
approved  ECPs-S. 

(12)  Nximber  of  SRS  requirements  modified  due  to 
approved  ECPs-S. 

(13)  Number  of  SRS  requirements  deleted  due  to 
approved  ECPs-S. 

e.  Frequency  of  reporting.  Monthly. 

f.  Use/ interpretation. 

(1)  When  a  program  begins,  the  details  of  its 
operation  and  design  are  rarely  complete,  so  it  is 
normal  to  experience  changes  in  the  specifications  as 
the  requirements  become  better  defined  over  time. 

(Note:  Rapid  prototyping  can  help  alleviate  this 
problem,  or  at  least  cause  refinement  to  happen  earlier 
in  development) .  When  design  reviews  reveal 
inconsistencies,  a  discrepancy  report  is  generated. 
Closure  is  accomplished  by  modifying  the  design  or  the 
requirements.  When  a  change  is  required  that  increases 
the  scope  of  the  project,  an  ECP-S  is  submitted. 

(2)  The  plot  of  open  discrepancies  can  be 
expected  to  spike  upward  at  each  review  and  to  diminish 
thereafter  as  the  discrepancies  are  closed.  For  each 
engineering  change,  the  amount  of  software  affected 
should  be  reported  in  order  to  track  the  degree  to 
which  ECPs-S  increase  the  difficulty  of  the  development 
fiffort.  Only  those  ECPs-S  approved  by  the 
configuration  control  board  should  be  counted.  Good 
requirements  stability  is  indicated  by  a  leveling  off 
of  the  cumulative  discrepancies  curve  with  most 
discrepancies  having  reached  closure. 
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(3)  Causes  of  program  turbulence  can  be 
investigated  by  looking  at  requirements  stability  and 
design  stability  together.  If  design  stability  is  low 
and  requirements  stability  is  high,  the  designer /coder 
interfaceis  suspect.  If  design  stability  is  high  and 
requirements  stability  is  low,  the  interface  between 
the  user  and  the  design  activity  is  suspect.  If  both 
design  stability  and  requirements  stability  are  low, 
both  the  interfaces  between  the  design  activity  and  the 
code  activity  and  between  the  user  and  the  design 
activity  are  suspect. 

(4)  The  metrics  for  requirements  stability 
should  be  used  in  conjunction  with  those  for 
requirements  traceability,  fault  profiles,  and  the 
optional  development  progress. 

(5)  Allowances  should  be  made  for  higher 
instability  in  the  case  where  rapid  prototyping  is 
utilized.  At  some  point  in  the  development  effort,  the 
requirements  should  be  firm  so  that  only  design  and 
implementation  issues  will  cause  further  changes  to  the 
specification. 

(6)  As  mentioned  previously,  it  is  recognized 
that  SLOC  is  somewhat  dependent  on  both  the  application 
language  as  well  as  the  style  of  the  programmer.  The 
key  is  to  watch  for  significant  changes  to  the  measure. 

g.  Rules  of  thumb.  No  formal  rules  are  given  here, 
but  a  high  level  of  instability  at  the  CDR  stage 
indicates  serious  problems  that  must  be  addressed  prior 
to  proceeding  to  coding.  For  MSCR  systems, 
requirements  stability  should  be  high  by  MS  II. 

h.  References.  Reference  (g)  of  Section  III  is 
applicable  to  this  metric. 

17-11.  Design  stability  metric 

a.  Purpose /description.  Design  stability  is  used 
to  indicate  the  amount  of  changes  made  to  the  design  of 
the  software.  The  design  progress  ratio  show  how  the 
completeness  of  the  design  is  advancing  over  time  and 
helps  give  an  indication  of  how  to  view  the  stability 
in  relation  to  the  total  projected  design. 

b.  Life  cycle  application.  Begin  tracking  no  later 
than  PDR  and  continue  for  each  version  until 
completion. 

c.  Algorithm/graphical  display. 

M  *  Number  of  modules  in  current 
delivery/design . 

Fg  «  Number  of  modules  in  current 

delivery/design  that  include  design 
related  changes  from  previous  delivery. 

Fj  »  Number  of  modules  in  current 

delivery/design  that  are  additions  to 
previous  delivery. 

Fjj  »  Number  of  modules  in  previous 

delivery/design  that  have  been  deleted. 
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Total  nodules  projected  for  project. 


S 

DP 


(stability)  »  [M  -  (F,  +  Fg  +  Fh)  ] 
(design  progress  ratio)  ■  M/T 


/  M 


(1)"  Although  not  indicated  in  Figure  l7-io,  it 
IS  possible  for  stability  to  be  a  negative  value.  This 
nay  indicate  that  everything  previously  delivered  has 
been  changed  and  nore  nodules  have  been  added  or 
deleted. 


Design  Stabilitv/Proaresa 


Stability  — Dealgn  Prograaa 


Pigura  17-10.  Sanpla  Dasign  Stabillty/Prograss  Matrie 


(2)  If  sone  nodules  in  the  current  delivery  are 
to  be  deleted  fron  the  final  delivery,  it  is  possible 
for  design  progress  to  be  greater  than  one. 

d.  Data  requirenents .  For  each  CSCI  and  each 
version: 


(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 


Date  of  conpletion 

M 

Fc 

F. 


S 

DP 


e.  Frequency  of  reporting, 
delivery. 


Monthly  or  at  each 
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f .  _  Use/ interpretation . 

(1)  Design  stability  should  be  monitored  to 
determine  the  number  and  potential  impact  of  design 
changes,  additions,  and  deletions  on  the  software 
configuration.  The  trend  of  design  stability  provides 
an  indication  of  whether  the  software  design  is 
approaching  a  stable  state,  that  is,  a  leveling  off  of 
the  curve  at  a  value  close  to  or  equal  to  one.  In 
addition  to  a  high  value  and  level  curve  the  following 
other  characteristics  of  the  software  should  be 
exhibited: 

(a)  The  development  progress  metric  is  high. 

(b)  Requirements  stability  is  high. 

(c)  Depth  of  testing  is  high. 

(d)  The  fault  profile  curve  has  leveled  off 
and  most  software  trouble  reports  (STRs)  have  been 
closed. 

(2)  Caution  must  be  exercised,  however,  due  to 
the  fact  that  this  metric  does  not  measure  the  extent 
or  magnitude  of  the  change  within  a  module. 

(3)  The  higher  the  stability,  the  better  the 
chances  of  a  stable  software  configuration.  However,  a 
value  close  to  one  is  not  necessarily  good  unless  M  is 
close  to  the  total  number  of  modules  required  in  the 
system  (DP  approaching  1) ,  and  the  magnitude  of  the 
changes  being  counted  are  relatively  small  and 
diminishing  over  time.  In  the  case  just  described, 
periods  of  inactivity  could  be  mistaken  for  stability. 

(4)  When  changes  are  being  made  to  the  software, 
the  impact  on  previously  completed  testing  must  be 
assessed. 

(5)  Allowance  for  exceptional  behavior  of  this 
metric  should  be  made  for  the  use  of  rapid  prototyping. 
It  is  thought  that  rapid  prototyping,  while  possibly 
causing  lower  stability  numbers  (i.e.,  higher 
instability)  early  in  the  program,  will  positively 
affect  the  stability  metric  during  later  stages  of 
development. 

(6)  Not  all  changes  made  to  the  software  are 
design  related. 

(7)  The  design  stability  metric  can  be  used  in 
conjunction  with  the  complexity  metric  to  highlight 
changes  to  the  most  complex  modules.  It  can  also  be 
used  with  the  requirements  metrics  to  highlight  changes 
to  modules  which  support  the  most  critical  user 
requirements. 

(8)  Finally,  the  design  stability  metric  does 
not  assess  the  quality  of  the  design.  Other  metrics 
(e.g.,  complexity)  can  contribute  to  such  an 
evaluation. 

g.  Rules  of  thumb. 

(1)  No  hard  and  fast  rules  are  known  at  this 
time.  However,  allowances  should  be  made  for  lower 
stability  in  the  case  of  rapid  prototyping  or  other 
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development  techniques  that  do  not  follow  the  standard 
waterfall  model  for  software  development.  In  either 
case,  an  upward  trend  with  a  high  value  for  both 
stability  and  design  progress  is  recommended  before 
acceptance  for  Government  testing.  A  downward  trend 
should  be-cause  for  concern. 

(2)  Experiences  with  similar  projects  should  be 
used  as  a  basis  for  comparison.  Over  time,  potential 
thresholds  may  be  developed  for  similar  types  of 
projects. 

h.  References.  Reference  (r)  of  Section  III  is 
applicable  to  this  metric. 

17-12.  Complexity  metric 

a.  Purpose/definition. 

(1)  Complexity  measures  give  an  indication  of 
the  structure  of  the  software  and  provide  a  means  to 
measure,  quantify  and/or  evaluate  the  structure  of 
software  modules.  They  also  indicate  the  degree  of 
unit  testing  which  needs  to  be  performed.  It  is 
commonly  believed  that  the  more  complex  the  software, 
the  harder  it  is  to  test  and  maintain.  Additionally,  a 
highly  complex  module  is  more  likely  to  contain 
embedded  errors  than  a  module  of  lower  complexity. 
Accordingly,  lower  complexity  ratings  reflect  software 
that  is  easier  to  test  and  maintain,  thus  logically 
resulting  in  fewer  errors  and  lower  life  cycle  costs. 

(2)  McCabe's  cyclomat ic  complexity  metric 
measures  the  internal  structure  of  a  piece  of  software. 

(3)  Halstead's  metrics  estimate  a  program's 
length  and  volume  based  on  its  vocabulary  (operators 
and  operands) . 

(4)  Other  simpler  complexity  metrics  are  control 
flow,  the  number  of  lines  of  code  per  module  (relates 
to  the  understandability  of  the  module) ,  and  the 
percent  comment  lines. 

b.  Life  cycle  application.  Begin  collecting 
McCabe's  cyclomat ic  complexity  metric  at  PDR.  Begin 
collecting  other  complexity  metrics  at  CDR,  as  the 
modules  are  placed  under  the  developer's  configuration 
control.  Revisit  during  PDSS  activities. 

.  c.  Algorithm/graphical  display.  Throughout  the 
discussion  of  complexity  measures,  the  term  "module"  is 
meant  to  be  equivalent  to  CSU. 

(1)  McCabe's  cyclomat ic  complexity  (C)  metric: 

C  -  E  -  N  +  2P,  where 

E  “  #  of  edges  (program  flows  between  nodes; 
i.e.  branches) 

N  “  #  of  nodes  (sequential  groups  of  program 
statements) 

P  *  #  of  connected  components  (number  of 
disconnected  parts  on  a  flow  graph) 
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w  1-  ^  example  of  a  module's  flow  orach  with 

eoBplexity  computed  ifshSw^l!; 
17—12  IS  a  histooram  of  a  CSCT's 
mo  ules  distributed  by  cyclomatic  complexity  value. 


**‘®  additional  ways  of  calculatino 
calculate  the  number  of^ 
ontrol  tokens  +  i.  Control  tokens  are  programmino 
language  statements  which  in  some  way  provide  decilion 

^he  top-down  flow  of  the  program 
In  other  words,  statements  such  as  IF,  GOTO,  CASE  ’ 

considered  to  be  control  tokens  since  they 
base  program  flow  upon  a  logical  decision  therebv  ^ 

folloi”^  l  program  execution  may 

ollow.  A  CASE  statement  would  contribute  (N  -  i)  to 

complexity,  where  N  is  the  number  of  conditions  ol 
cases  associated  with  the  statement. 

(4)  Halstead's  metrics: 


Vocabulary:  v  «  n,  +  n. 

Program  Length:  L  «  nT  +  N, 

Velum.:  V  -  LCl^jV),  where 

=  #  distinct  operators 
n2  «  #  distinct  operands 
Ni  =  total  #  occurrences  of  the  operators 
Nj  -  tota^  0  occurrences  of  the  operands 
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McCabe  Cyclomatic  Complexity 


Complexity 

■  CSC!  3  (200  modules} 

Figura  17-12.  saapla  Conplaxity  Matric 


(5)  Control  flow  metric: 

Count  the  number  of  times  in  each  module 
where  control  paths  cross.  Control  path  crossings  are 
also  referred  to  as  "Jcnots”.  For  example,  a  GOTO 
statement  would  cause  a  Jcnot  to  occur  in  a  module's 
flow  graph. 

Non-comment,  non-blanJc,  executable,  and  data 
statements  (herein  referred  to  as  lines  of  code  (LOC) ) 
per  module  metric:  ' 

Count  the  number  of  non-blank,  non-comment, 
executable,  and  data  statements  in  each  module. 

(7)  The  percent  comment  lines  metric  shows  the 
relative  amount  of  explanatory  material  in  a  module 
compared  to  its  size,  in  lines.  For  the  purposes  of 
computation,  blank  lines  are  not  considered  comment 
X  xn6s • 


Percent  comment  lines  metric: 

Percent  comment  lines  ■  (C  /  T)  *  lOO,  where 

C  »  #  comment  lines  in  module 
T  «  total  #  non-blank  lines  in  module 

d.  Data  requirements:  For  each  module: 

(1)  Source  code  or  PDL 

(2)  Programming  language 
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(3)  Number  of  nodes 

( 4 )  Number  Of  edges 

(5)  Number  of  connected  components 

(6)  Number  of  distinct  operators 

(7)  Number  of  distinct  operands 

(8) .  Total  number  of  occurrences  of  the  operators 

(9)  Total  number  of  occurrences  of  the  operands 

(10)  Number  of  times  in  each  module  where 
control  paths  cross 


Number  of  LOCs  (as  defined  above)  per  module 

(12)  Percent  comment  lines 

(13)  All  complexity  input  data  and  results  of 
calculations  listed  above 

of  reporting.  Monthly,  for  any  module 
that  has  been  modified. 


f.  Use/ interpretation. 

(1)  Automated  tools  are  available  and  should  be 
used  to  assist  in  the  computation  of  the  complexity 
measures.  * 


(2)  This  metric  is  used  throughout  the  software 
life  cycle.  Requiring  a  complexity  limit  as  a 
contractual  requirement  will  stimulate  structured 
programming  techniques,  thereby  impacting  desion 
limiting  the  number  of  basis  paths  (i.e.  critics 
paths)  in  a  program  at  the  design  and  coding  stages. 

It  IS  used  during  software  testing  to  identify  basis 
paths,  to  define  and  prioritize  the  testing  effort,  and 
to  assess  the  completeness  of  CSU  testing.  During  the 
maintenance  phase,  a  proposed  change  should  not  be 
allowed  to  substantially  increase  the  complexity,  as 
this  could  increase  the  testing  effort  and  decrease 
maintainability. 

(3)  The  complexity  metrics  should  be  generated 
for  each  module  in  the  system.  The  metrics  can  be 
grouped  for  display  in  a  number  of  ways  (e.g.  by  CSCI, 
by  individual  CSU,  etc.).  Examination  at  various 
levels  can  provide  indications  of  potential  problem 
areas.  These  indications  can  give  guidance  to  the 
developer  on  areas  where  additional  concentration  is 
needed,  as  well  as  areas  where  the  Government  test 

should  focus,  such  as  code  waDcthroughs,  more 
comprehensive  unit  level  testing,  or  stress  testing. 
Figure  17-12  portrays  a  snapshot  in  time  of  the 
complexity  values  for  all  the  modules  in  a  given  CSCI. 

'the  majority  of  them  are  within  the  range  of  the 
rules  of  thumb,  it  can  be  seen  that  several  of  them 
have  well  exceeded  them.  These  modules  should  be 
further  scrutinized  through  testing  and  analysis. 

(4)  Examination  of  complexity  trends  over  time 
can  also  provide  useful  insights,  especially  when 
combined  with  other  metrics  such  as  design  stability  or 
the  optional  development  progress.  For  example,  late 
software  code  "patches”  may  cause  the  complexity  of  the 
patched  module  to  exceed  an  acceptable  limit. 
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indicating  that  the  design  rather  than  the  code  should 
have  been  changed.  It  is  noted  that  test  resources  are 
better  expended  on  modules  that  have  a  relatively  high 
structural  complexity  rather  than  on  software  that  will 
reflect  a  high  number  of  lines  of  code  tested. 

(5)  _  Wherever  possible,  complexity  should  be 
computed  for  the  PDL.  The  section  on  rules  of  thumb 
describes  special  interpretation  factors  for  the 
complexity  of  PDL. 

(6)  There  are  several  embedded  assumptions  and 
known  weaknesses  in  the  complexity  metrics.  For 
example,  in  the  computation  of  McCabe's  cyclomatic 
complexity,  there  is  no  differentiation  between 
different  kinds  of  control  flows.  A  CASE  statement 
(which  is  easier  to  use  and  understand  than  the 

•  corresponding  series  of  conditional  statements)  makes  a 
high  contribution  to  complexity,  which  is  somewhat 
counterintuitive  when  one  considers  that  the 
corresponding  series  of  IF. .THEN. .ELSE  statements  would 
probably  be  oore  troublesome  from  the  standpoint  of 
testing,  modification,  and  maintenance.  Further,  a 
million  straight  line  instructions  are  judged  to  be  as 
complex  as  a  single  instruction.  Additionally,  the 
interpretation  of  complexity  will  be  different  for 
different  higher  order  languages. 

(7)  There  are  many  ways  of  defining  and  counting 
lines  of  code.  The  fairly  simple  definition  given  in 
the  algorithm  above  is  intended  to  apply  somewhat 
equally  across  the  spectrum  of  procedural  languages. 

It  should  be  noted  that  much  research  is  ongoing  into 
methods  and  tools  for  counting  lines  of  code  for  each 
language.  The  definition  given  here,  and  used  as  well 
in  some  of  the  other  metrics,  is  intended  to  indicate 
relative  amounts  of  change  in  modules  as  they  are  built 
and  maintained,  not  just  size.  This  definition  may  at 
least  allow  historical  comparisons  to  be  done  on 
similar  implementations. 

(8)  It  is  also  recognized  that  the  percent  of 
comment  lines  is  a  very  language  dependent  measure. 
Additionally,  the  metric  does  not  address  the 
usefulness  or  completeness  of  the  comments.  Some  self- 
documenting  languages  require  fewer  comments  than  an 
assembly  language  or  a  language  like  FORTRAN. 

(9)  It  is  acknowledged  as  well  that  the 
complexity  metrics  as  outlined  in  this  pamphlet  are 
oriented  towards  procedural  programming  languages. 

When  applied  to  artificial  intelligence  and  pure  object 
oriented  languages,  care  must  be  taken  in  interpreting 
the  results. 

(10)  In  cases  of  high  module  cyclomatic 
complexity,  various  means  exist  to  help  identify  how  it 
may  be  reduced.  These  techniques  include  calculation 
of  actual  complexity  and  essential  complexity.  For 
further  details  see  references  cited  in  part  (h) . 
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(11)  The  use  of  additional  complexity  or  design 
structure  metrics  for  languages  li)ce  Ada  (to  measure 
the  degree  of  encapsulation,  for  example)  can  lend 
additional  insights  into  the  software  structure.  Their 
use  is  encouraged,  but  not  required  via  this  pamphlet. 

(12) .  It  is  recommended  that  this  metric  be  used 
as  soon  as  practical  (e.g.,  as  code  is  being 
developed) .  Also,  it  must  not  be  relied  upon  as  the 
sole  metric  to  judge  the  quality  of  the  design's 

imp 1 emen t a t ion. 

(13)  Additional  research  is  underway  on  ways  of 
assessing  the  complexity  of  a  design  before  any  code  is 

As  these  design  complexity  metrics  evolve, 
their  use  should  be  pursued,  as  it  is  highly  desirable 
to  limit  the  inherent  complexity  of  software  while 
still  in  the  design  phase. 

(14)  To  trade  off  the  benefits  and  limitations 
of  each  of  the  individual  complexity  measures  discussed 
in  this  section,  the  complexity  measures  should  be  used 
as  a  group.  They  also  should  be  evaluated  in 
conjunction  with  the  metrics  for  fault  profiles  and 
depth  of  testing. 

g.  Rules  of  thumb. 

(1)  The  following  thresholds,  which  should  be 
applied  at  the  module  level,  are  widely  accepted  as 
industry  standards.  Any  value  not  meeting  these~'»’P'^ 
thresholds  should  cause  concern  about  potential 
problems  in  the  specific  metric  area. 


Metpig  Threshold  (per  module) 

Cyclomatic  complexity  <«  lo 

Control  flow  «  0 

Volume  <K  3200 

LOC  <«  200 

%  Comment  lines  »  60  % 


(2)  For  cyclomatic  complexity,  the  suggested 
limit  is  ten  for  any  module.  The  references  below  show 
that  the  error  rate  and  number  of  bugs  observed  for 
modules  having  complexities  of  less  than  ten  is 
substantially  lower  than  those  with  complexities 
greeter  than  ten.  A  module  with  complexity  greater 
than  ten  may  need  to  be  restructured  (if  feasible)  into 
several  less  complex  ones,  if  the  complexity  is  due  to 
structures  li)ce  CASE  statements,  the  complexity  can  be 
accommodated.  However,  if  the  high  complexity  is  due 
to  structures  like  DO  loops  and  other  raw  logic, 
serious  attempts  should  be  made  to  redesign  or 
subdivide  the  module. 

(3)  A  more  recent  approach  varies  the  threshold 
for  cyclomatic  complexity  according  to  life  cycle 
phase.  During  the  design  phase  it  is  suggested  that 
the  value  not  exceed  seven  for  program  design  language 
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to  allow  for  expected  growth  to  a  value  of  ten  during 
implementation  to  code. 

h.  References.  References  (d) ,  (1),  (n) ,  (p) ,  (q) , 
(r)  and  (v)  of  Section  III  are  applicable  to  this 
metric. 

17-13.  Breadth  of  tastiag  aatrie 

a.  Purpose/description.  Breadth  of  testing 
addresses  the  degree  to  which  required  functionality 
has  been  successfully  demonstrated  as  well  as  the 
amount  of  testing  that  has  been  performed.  This 
testing  can  be  called  "blac)c  box”  testing,  since  one  is 
only  concerned  with  obtaining  correct  outputs  as  a 
result  of  prescribed  inputs. 

b.  Life  cycle  application.  Begin  collecting  data 
at  the  end  of  unit  testing. 

c.  Algorithm/graphical  display. 

(1)  Breadth  of  testing  consists  of  three 
different  measures.  One  measure  deals  with  coverage 
and  two  measures  deal  with  success.  These  three 
subelements  are  portrayed  in  the  following  equation: 

CoTsrag*  T«st  Suecasi  Orarall  Suecass 

#  raqts  tested  #  reqts  passed  #  reqts  passed 

-  *  -  m  - 

total  #  reqts  #  reqts  tested  total  #  reqts 

(2)  Breadth  of  testing  "coverage”  is  computed  by 
dividing  the  number  of  requirements  that  have  been 
tested  (with  all  applicable  test  cases  under  both 
representative  and  maximum  stress  loads)  by  the  total 
number  of  requirements. 

(3)  Breadth  of  testing  "test  success”  is 
computed  by  dividing  the  number  of  requirements  that 
have  been  successfully  demonstrated  through  testing  by 
the  number  of  requirements  that  have  been  tested. 

(4)  Breadth  of  testing  "overall  success”  is' 
computed  by  dividing  the  number  of  requirements  that 
have  been  successfully  .demonstrated  through  testing  by 
the  total  number  of  requirements. 

(5)  All  three  measures  of  breadth  of  testing 
should  be  trac)ced  for  SRS  requirements,  IRS 
requirements  and  UFD  requirements,  throughout  the 
development  process  if  possible.  In  actuality, 
functional  testing  per  CSCI  is  normally  only  done  as 
part  of  FQT  and  system  level  testing. 

(€)  The  results  of  each  measure  must  be  reported 
and  can  be  simultaneously  displayed  over  Jcey  test 
events  with  the  use  of  stac)ced  vertical  bar  graphs  as 
shown  in  Figure  17-13.  Coverage  and  success  can  also 
be  displayed  over  time  in  a  manner  similar  to  that 
shown  in  Figure  17—14,  for  the  depth  of  testing  metric. 
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Breadth  of  Testinn 


Percent  UFD  Reqts  -  CSCI  2 


■  Test  Coverage  Elest  Success  B  Overall  Success 


Pigura  17-13.  saapla  Braadth  of  Tasting  Natrie 


(7)  It  is  suggested  that  coverage  or  success 
values  be  expressed  as  percents  by  multiplying  each 
value  by  lOO  before  display  in  order  to  facilitate 
understanding  and  commonality  among  metrics 
presentations . 

d.  Data  requirements.  For  each  CSCI: 

(1)  Number  of  SRS  requirements. 

(2)  Number  of  SRS  requirements  tested  with  all 
planned  test  cases. 

(3)  Number  of  SRS  requirements  successfully 
demonstrated  through  testing. 

(4)  Nximber  of  IRS  requirements. 

(5)  Number  of  IRS  requirements  tested  with  all 
planned  test  cases. 

(6)  Number  of  IRS  requirements  successfully 
demonstrated  through  testing. 

(7)  For  each  of  the  four  UFD  priority  levels: 

(a)  Number  of  UFD  requirements. 

(b)  Number  of  UFD  requirements  tested  with 
all  planned  test  cases. 

(c)  Number  of  UFD  requirements  successfully 
demonstrated  through  testing. 

(8)  Test  identification  (e.g.,  FQT,  DT,  OT) . 

e.  Frequency  of  reporting.  Monthly  throughout* 
software  functional  and  system  level  testing. 
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f.  Use/ interpretation. 

coverage  portion  of  breadth  of  testing 
^^®  o^  testing  performed  without 

success.  By  observing  the  trend  of  coverage 
over  time,  one  gets  an  idea  of  the  extent  of  full 
testing  that  has  been  performed. 

J^®  f^ccoss  portion  of  breadth  of  testing 
indications  about  requirements  that  have  been 
demonstrated.  By  observing  the  trend  of 
the  overall  success  portion  of  breadth  of  testing  over 
time,  one  gets  an  idea  of  the  growth  in  successfully 
demonstrated  functionality.  ^ 

testing  metrics  for  coverage 
and  overall  success  should  be  used  together  and  in 
conjunction  with  the  requirements  traceability  metrics, 
the  (optional)  development  progress  metrics,  and  fault 
so  that  potential  problem  areas  can  be 
Identified.  The  breadth  of  testing  metric  must  be  used 
in  conjunction  with  the  metrics  for  depth  of  testing, 
requirements  stability,  arid  design  stability. 

TTT.r^  •  ^^2  most  innovative  aspects  of  the 

UFD  is  the  categorization  of  requirements  in  terms  of 
levels.  With  this  approach,  the  most 
requirements  can  be  highlighted.  Using  this 
prioritization  scheme,  one  can  partition  the  breadth  of 
resting  metric  to  address  each  priority  level.  At 
various  points  along  the  development  path,  the  pivotal 
requirements  for  that  phase  can  be  addressed  in  terms 
^r*®ing,  test  coverage,  and  test  success. 

V.  ®®®^  priority  level,  the  test  cases 

for  evaluating  the  success  of  a  requirement 
should  be  developed  through  the  Test  Integration 
Working  Group  (TIWG)  process  in  order  that  sufficient 

requi?emlntr®  ^®"®^®^®‘^  adequately  demonstrate  the 

(6)  Failing  only  one  test  case  results  in  a 
requirement  not  being  successfully  demonstrated.  If 

resources  exist,  an  additional,  optional  way 
o  a  dress  breadth  of  testing  consists  of  examining 
each  requirement  in  terms  of  the  percent  of  test  clses 
that  have  t>een  performed  and  passed.  In  this  way, 
partial  credit  for  testing  a  requirement  can  be  shown 

®®®®?  ®^^®^  ®  requirement), 

Shich^?.  nothing  approach.  This  method, 

which  IS  not  mandated  here  due  to  cost  and  reporting 

considerations,  may  be  useful  in  providing  additional 
granularity  into  breadth  of  testing. 

w  .  It. is  recognized  that  there  is  possible 
subjectiyity  in  assessing  whether  requirements  have 
been  satisfied. 

(8)  It  should  be  noted  that  some  requirements 
may  not  be  testable  until  very  late  in  the  testing 
process  (if  at  all) .  ^ 
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(9)  Breadth  of  testing  should  be  revisited  as  a 
result  of  Government  testing.  Also,  any  time  there  is 
a  change  in  requirements,  breadth  of  testing  must  be 
revisited. 

(10)  During  post  deployment  software  support, 
this  metric  should  be  used  with  the  requirements 
traceability  metrics  to  indicate  areas  which  need 
regression  testing. 

g.  Rules  of  thumb.  The  Government  should  clearly 
specify  what  functionality  should  be  in  place  at  each 
phase  of  development.  This  process  is  obviously  system 
dependent.  At  each  stage  of  testing  (unit  through 
system  level  stress  testing) ,  emphasis  should  be  placed 
on  demonstrating  that  a  high  percentage  of  the 
functionality  needed  for  that  stage  of  testing  is 
achieved.  Prior  to  the  formal  Government  operational 
test,  most  functions  should  be  demonstrated  under 
stress  loading  conditions. 

h.  References.  Reference  (o)  of  Section  III  is 
applicable  to  this  metric. 

17-14.  Depth  of  testing  metric 

a.  Purpose/description.  The  depth  of  testing 
metric  provides  indications  of  the  extent  and  success 
of  testing  from  the  point  of  view  of  coverage  of 
possible  paths/conditions  within  the  software.  Depth 
of  testing  consists  of  three  separate  measures  (plus  an 
optional  fourth) ,  each  of  which  is  comprised  of  one 
coverage  and  two  success  sub-elements  (similar  to 
breadth  of  testing) .  The  testing  can  be  called  "white 
box"  testing,  since  there  is  visibility  into  the 
paths/conditions  within  the  software. 

b.  Life  cycle  application.  Begin  collecting  data 
at  CDR  and  continue  through  development  as  changes 
occur  in  either  design,  implementation  or  testing. 
Revisit  as  necessary  during  PDSS. 

c.  Algorithm/graphical  display.  A  path  is  defined 
as  a  logical  traversal  of  a  module,  from  an  entry  point 
to  an  exit  point.  If  one  refers  bacJc  to  the  discussion 
of  the  complexity  metric,  a  path  is  actually  a 
combination  of  edges  and  nodes.  See  the  complexity 
metric  for  the  definition  of  an  edge. 

(1)  The  path  metric  for  each  module  is  defined 
as  the  number  of  paths  in  the  module  that  have  been 
successfully  executed  at  least  once,  divided  by  the 
total  niunber  of  paths  in  the  module. 

(2)  The  statement  metric  for  each  module  is rthe 
number  of  executable  statements  in  the  module  that  have 
been  successfully  exercised  at  least  once,  divided  by 
the  total  number  of  executable  statements  in  the 
module. 

(3)  The  domain  metric  for  each  module  is  the 
number  of  input  instances  that  have  been  successfully 
tested  with  at  least  one  legal  entry  and  one  illegal 
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entry  in  every  field  of  every  input  parameter,  divided 
by  the  total  number  of  input  instances  in  the  module. 
(Note:  In  addition  to  test  coverage,  this  metric  serves 
to  partially  address  the  robustness  of  the  software.) 

(4)  The  optional  decision  point  metric  for  each 
module  is .the  number  of  decision  points  in  the  module 
that  have  been  successfully  exercised  with  all  classes 
of  legal  conditions  as  well  as  one  illegal  condition 
(if  any  exist)  at  least  once,  divided  by  the  total 
number  of  decision  points  in  the  module.  Each  decision 
point  containing  an  "or"  should  be  tested  at  least  once 
for  each  of  the  condition's  logical  predicates. 

(5)  Figure  17-14  portrays  test  coverage  and 
overall  success  for  the  paths  within  a  CSCI,  which  is 
computed  as  a  composite  of  its  module  level  path 
metrics.  The  other  aspects  of  depth  of  testing  can  be 
illustrated  in  a  similar  fashion.  See  the  breadth 
oftesting  metric  for  details  on  computing  coverage  and 
success  values. 


Depth  of  Testing 


Percent  of  Paths  -  CSCI  3 


4  5  6  7  8  9  10  11  12 

Program  Month 

ICoserage  QOvtrall  Success 


Figure  17-14.  Sample  Depth  of  Testing  Metric 

d.  Data  recpiirements :  For  each  nodule: 

(1)  CSCI  identifier 

(2)  CSC  identifier 

(3)  CSU  identifier 

(4)  Number  of  paths 

(5)  Number  of  statements 

(€)  Number  of  input  instances 

(7)  Number  of  paths  tested 

(8)  Number  of  statements  tested 
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(9)  Number  of  input  instances  tested 

(10)  Number  of  decision  points  tested 

(11)  Number  of  paths  which  have  been 
successfully  executed  at  least  once 

(12)  Number  of  statements  that  have  been 
successfully  exercised 

(13)  Number  of  inputs  which  have  been 
successfully  tested  with  one  legal  entry 
and  one  illegal  entry 

(14)  Number  of  decision  points  (optional) 

(15)  Number  of  decision  points  which  have 
been  successfully  exercised  at  least 
once  with  all  legal  classes  of  condi¬ 
tions  and  one  illegal  condition  (opt.) 

e.  Frequency  of  reporting.  Monthly.  Report  only 
those  modules  (CSUs)  that  have  been  modified  or  further 
tested  after  their  depth  of  testing  values  have  been 
reported  for  the  first  time.  Note:  In  recognition  of 
the  effort  required  to  collect  and  report  this  metric, 
the  following  rules  are  offered.  Always  compute  the 
domain  metric  as  it  is  relatively  straightforward. 
Compute  the  path  and  statement  metrics  if  automated 
tools  exist.  If  no  automated  tools  exist,  compute 
these  two  metrics  if  the  module  implements  a  UFD 
priority  one  requirement,  or  if  the  complexity  rules  of 
thumb  are  exceeded  for  that  module. 

f .  Use/ interpretation . 

(1)  The  depth  of  testing  metric  attacks  the 
issue  of  test  coverage,  test  success,  and  overall 
success  by  considering  the  paths,  statements,  inputs 
and  decision  points  of  the  software.  The  trend  of 
these  depth  of  testing  metrics  over  time  provides 
indications  of  the  progress  of  successful  testing,  and 
also  test  sufficiency. 

(2)  The  metrics  are  collected  at  the  module  or 
CSU  level,  but  they  can  be  easily  extended  to  the  CSC, 
CSCI,  or  system  level  by  simply  replacing  the  term 
"module"  in  the  algorithm  definition  above  with  the 
term  "CSC,"  or  -^CSCI,"  or  "system".  Early  in  the 
contractor  testing  process,  it  makes  more  sense  to 
assess  depth  of  testing  at  the  module  or  CSU  level,  but 
later  it  makes  more  sense  to  consider  CSCs  and  CSCIs. 

(3)  The  depth  of  testing  metric  can  be  used  to 
focus  attention  on  those  modules  which  implement  high 
priority  UFD  requirements. 

(4)  The ^ depth  of  testing  metric  measures  should 
be  used  in  conjunction  with  requirements  traceability, 
fault  profiles,  complexity,  and  the  optional 
development  progress  metrics.  For  example,  with 
complexity,  th^j  modules  of  highest  complexity  could  be 
highlighted  for^testing  (e.g.,  one  might  want  to  test 
those  modules  first) .  They  must  be  used  with  the 
breadth  of  testing  metrics  to  insure  that  all  aspects 
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approaching  an  acceptable  state  for  the 

Governinent  • 

g.  Rules  of  thumb.  None. 

(o)  of  Section  III  is 

applicable  to  this  metric. 

17-15.  Fault  profiles  metric 

profiles  provide 

insight  into  the  quality  of  the  software,  as  well  as 

to  fix  knovm  faults.  Note 
^5®®®  actually  come  from  measuring  the 

-faults-)  in  the  software.  Early 
in  the  development  process,  fault  profiles  can  be  used 
to  measure  the  quality  of  the  translation  of  the 
software^ requirements  into  the  design.  Later,  they  can 
be  used  to  measure  the  quality  of  the  implementation  of 
the  software  recjuirements  and  design  into  code. 

r>^  application.  Begin  after  completion 

of  unit  testing  when  software  has  been  brought  under 
configuration  control  by  the  developer.  Continue 
through  PDSS. 

c.  Algorithm/graphical  display. 

(1)  Plot  the  cumulative  number  of  detected  and 
cumulative  nujnber  of  closed  software  faults  as  a 

K  time,  as  shown  in  Figure  17-15.  One  plot 

be  developed  for  each  priority  level  as  defined 
in  the  data  requirements  section  below. 

(2)  Plot  the  number  of  software  faults  detected 
and  software  faults  closed  per  month.  This  is 
illustrated  in  Figure  17-16. 

(3)  Histograms  of  open  faults  by  CSCI  and 
priority  can  be  portrayed  as  in  Figure  17-17. 

Relative  CSCI  status  with  respect  to  open 
faults  can  be  shown  as  .in  Figure  17-18. 

(5)  Calculate  the  average  age  of  STRs  as 

follows;  For  all  STRs,  sum  the  days  between  the  time 
th®  current  date  (if  still  open).  Divide  this  sum  by 
th®  fault  was  opened  and  either  when  it  was  closed  or 
the  total  number  of  STRs.  This  can  be  displayed  as 
shown  in  Figure  17-19.  *  ^  ^ 

(6)  Calculate  the  average  age  of  closed  faults 

closed  STRs,  sum  the  days  from  the 
time  the  STR  was  opened  and  when  it  was  closed.  Divide 
this  by  the  total  number  of  closed  STRs.  This  can  be 
calculated  for  each  problem  priority  or  overall 

requirements.  The  following  informition 
should  be  derived  from  all  software  trouble  reports. 
These  data  will  be  used  to  calculate  higher  level 
statistics  (described  in  later  paragraphs)  as  well  as 
to  support  queries  generated  as  a  result  of  the 
graphical  representations. 

(1)  Unique  identifier 

(2)  Date  written 

(3)  Descriptive  title  of  problem 
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CSCI  Open  Aae 

(faults  still  open) 


Avg  Tim*  Op*n  (w**k*) 


■H  Priority  1 
S3  Priority  4 


^ZZl  miorltv  2 
GS  Priority  S 


Pigura  17-17 .  Saapla  of  Avaraga  opan  Aga  of  STRa 


Average  Age  of  STRs 

(open  only) 

Average  Age  of  STRs  (weeks) 


-^Average  Age  of  STRs 

Figure  17-18.  Saapla  of  Avaraga  Aga  of  Opan  Faults 
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Average  Aoe  nf  .^trq 
(open  and  closed) 


Pigura  17-19.  8aBpla  of  Avaraga  Aga  of  All  8TRa 

(4)  Detailed  description  of  problem  (optional) 

(a)  1  «  causes  mission  essential  function  (or 

operator  s  accomplishment  thereof)  to  be  disabled  or 
jeopardizes  personnel  safety. 

2  *  causes  mission  essential  function  (or 
operator  s  accomplishment  thereof)  to  be  degraded. 

There  is  no  work  around.  ««yiraaea. 

(c)  3  »  causes  mission  essential  function  (or 

operator's  accomplishment  thereof)  to  be  degraded?  ^ 
There  is  a  reasonable  work  around. 

4.^  *  causes  operator  inconvenience  but 
doesn  t  affect  a  mission  essential  function. 

(®)  5  •  all  other  errors 

(6)  Category: 

(a)  Requirements 

(b)  Design 

(c)  Code 

(d)  Documentation 

(e)  Other 

(7)  When  discovered: 

(a)  Requirements  analysis 

(b)  Design  review 

(c)  Code  and  unit  test 

(d)  Integration  and  test 

(e)  Operation/maintenance 
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(8)  Status: 

(a)  Open 

(b)  Duplicate 

(c)  Closed 

(d)  Invalid 

(9)  .  Date  detected 

(10)  Date  closed 

(11)  Software  module  (CSU) 

(12)  CSCI 

(13)  Software  version 

(14)  Effort  to  fix  (man-hours) 

(15)  The  following  rolled-up  statistics  are  the 
building  bloc)cs  for  the  graphical  representations  of 
fault  profiles.  For  each  CSCI,  for  each  priority: 

(a)  Cumulative  number  of  STRs 

(b)  Cumulative  number  of  closed  STRs 

(c)  Average  age  of  closed  STRs 

(d)  Average  age  of  open  STRs 

(e)  Average  age  of  STRs  (both  open  and 

closed) 

(f)  Total  number  of  problems  for  each 
category  (described  above) 

e.  Frequency  of  reporting.  Monthly. 

f.  Use/ interpretation. 

(1)  There  are  various  aspects  of  fault  profiles 
that  can  be  examined  for  insights  into  quality 
problems.  The  most  popular  type  of  graphical 
representation,  portrayed  in  Figure  17-15,  displays 
detected  faults  and  closed  (corrected  and  verified) 
faults  on  the  same  scale.  These  types  of  graphs  should 
be  examined  for  each  problem  priority  level,  and  for 
each  major  module  or  CSCI.  Applied  during  the  early 
stages  of  development,  fault  profiles  measure  the 
^aiity  of  the  translation  of  the  software  requirements 
into  the  design.  STRs  opened  during  this  phase  suggest 
that  requirements  are  not  being  defined  correctly. 
Applied  later  in  the  development  process,  assuming 
adequate  testing,  fault  profiles  measure  the 
implementation  of  requirements  and  design  into  code. 
STRs  opened  during  this  stage  could  be  the  result  of 
having  an  inadequate  design  to  implement  those 
requirements,  or  a  poor  implementation  of  the  design 
into  code.  An  examination  of  the  fault  category  should 
provide  indications  of  these  causal  relationships. 

These  examination  should  be  performed  as  a  matter  of 
course  in  any  analysis  of  fault  profiles.  One  should 
continuously  observe  the  gap  between  open  and  closed 
faults;  if  a  constant  gap  or  a  continuing  divergence  is 
observed,  especially  as  a  Jcey  test  or  milestone  is 
approached,  appropriate  action  should  be  taJcen. 

(2)  Another  use  of  fault  profiles  consists  of 
monthly  non-cumulative  total  for  each  CSCI  and  priority 
(Figure  17-16) .  This  can  be  compared  to  the  amount  of 
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testing  done  in  those  months  to  provide  insights  into 
the  adequacy  of  the  test  program. 

(3)  Open  age  histograms  (Figures  17-17  and  17- 
18)  can  be  used  to  indicate  which  CSCIs  and  which 
priorities  are  the  most  troublesome  with  respect  to 
fixing  faiilts.  This  may  serve  to  indicate  that  the 
developing  group  for  that  CSCI  may  need  assistance, 
whether  due  to  a  difficult  set  of  requirements  or  for 
some  other  cause. 

(4)  Average  age  graphs  (Fi^re  17-19)  can  track 
whether  the  time  to  close  faults  is  increasing  with 
time,  which  may  be  an  indication  that  the  developer  is 
becoming  saturated  or  that  some  faults  are  exceedingly 
difficult  to  fix. 

(5)  Caution  must  be  used  in  interpreting  the 
fault  profiles,  as  the  detection  of  errors  is  closely 
tied  to  the  quality  of  the  development  and  testing 
process.  That  is,  a  low  number  of  detected  faults 
could  indicate  a  good  product  from  a  good  process  or 
simply  a  bad  process  to  start  with  (e.g.,  one  with 
inadequate  testing) .  Conversely,  a  large  number  of 
faults  early  on  in  a  program  may  not  be  bad.  For 
example,  the  developer  may  have  an  aggressive  testing 
program  or  may  be  using  techniques  such  as  rapid 
prototyping  with  a  heavy  user  involvement  to  wring  out 
requirements  early.  A  large  number  of  STRs  opened  in  a 
particular  month  may  be  the  result  of  errors  detected 
during  a  specification  review,  audit,  test,  or  from  use 
of  the  software  in  the  field.  Thus,  the  measures 
cannot  be  assessed  without  also  considering  the 
measures  on  breadth  and  depth  of  testing.  The  fault 
profiles  should  also  be  used  in  conjunction  with  the 
metrics  for  complexity,  design  stability,  and 
requirements  stability. 

(6)  If  the  cumulative  number  of  closed  STRs 
remains  constant  over  time  and  a  number  of  STRs  remain 
open,  this  may  indicate  a  lack  of  problem  resolution. 
The  age  of  the  open  STRs  should  be  checked  to  see  if 
they  have  been  open  for  an  unreasonable  period  of  time. 
If  so,  these  STRs  represent  areas  of  increased  risk. 

The  cause  for  lack  of  resolution  needs  to  be  idi^tified 
and  corrective  action  taken.  -,.h 

(7)  Once  an  average  STR  age  has  been  established 
large  individual  deviations  should  be  investigated. 
There  are  several  reasons  why  STRs  may  remain  open  for 
a  lengthy  period  of  time.  One  reason  could  be  that  the 
STR  is  a  result  of  identification  of  an  inadequate 
requirement  which  needs  to  be  refined  and  is  undergoing 
review.  An  ECP-S  may  have  been  written  for  a  problem 
noted  and  is  waiting  res^olution.  It  could  also  mean 
that  the  contractor  has  failed  to  take  corrective 
action  on  the  problem.  Again,  the  reasons  for  lack  of 
problem  resolution  need  to  be  identified  and  corrective 
action  taken.  The  average  open  age  of  high  priority 
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faults  should  also  be  examined  with  respect  to  the  time 
remaining  to  the  next  major  test  or  milestone.  If  the 
average  age  of  open  high  priority  faults  exceeds  the 
time  remaining,  consideration  should  be  given  to 
delaying  the  test  or  milestone  until  the  problems  are 
resolved.  . 

(8)  As  an  option  the  following  method  of 
assessing  test  adequacy  can  be  used.  Rome  Air 
Development  Center  (RAOC)  conducted  wide  research  on 
several  software  projects  and  identified  the 
characteristics  which  have  direct  impact  on  system 
reliability.  RADC-TR-87-171  Volumes  I  and  II, 
"Methodology  for  Software  Reliability  Prediction"  can 
be  used  to  predict  the  number  of  faults  expected  to  be 
present  per  configuration  item.  Using  this  information 
as  a  guideline,  fault  profiles  can  be  compared  to  these 
estimates  to  determine  if  the  "peak  level"  of  opened 
STRs  is  being  approached  and  that  the  software 
development  process  has  matured.  As  an  example, 
suppose  the  predicted  number  of  faults  is  much  higher 
than  the  actual  number  of  faults  reported.  If  the  test 
coverage  metrics  are  low,  this  suggests  that  testing  is 
not  complete  and  the  remaining  faults  are  yet  to  be 
found.  If  the  test  coverage  is  high  this  could  mean 
that  the  software  was  well  written  to  start  with,  or 
that  the  test  cases  used  were  not  thorough  enough.  If 
the  prediction  is  much  less  than  the  actual  number  of 
faults,  this  may  indicate  a  niunber  of  problems.  The 
software  developer  may  have  an  inadequate  development 
effort,  may  have  encountered  unexpected  design 
difficulties,  have  faulty  coding  or  an  especially 
troublesome  module (s).  The  requirements  stability 
metric  should  be  checked  and  if  it  is  high,  this  would 
indicate  an  immature  baseline. 

g.  Rules  of  th\imb. 

(1)  The  Government  should  not  accept  software 
for  any  formal  system  level  Government  testing  until, 
at  a  minimum,  all  priority  one  and  two  faults  have  been 
closed.  Furthermore,  a  large  number  of  lower  priority 
faults  should  be  examined  for  a  possible  cumulative 
effect  on  successful  test  conduct. 

(2)  If  tracking  the  fault  profiles  starts  early 
in-  software  development,  an  average  STR  open  age  of 
less  than  three  months  may  be  experienced.  After 
fielding,  this  value  can  rise  primarily  due  to  the 
necessary  delays  typically  experienced  in  the  PDSS 
process,  such  as  scheduled  block  releases. 

h.  References.  References  (c) ,  (h) ,  (i) ,  and  (t) 
of  Section  III  are  applicable  to  this  metric. 

17-16.  Reliability  metric 

a.  Purpose /description. 

(1)  One  aspect  of  the  reliability  metric  is  to 
assess  the  contribution  of  software  to  the  overall 
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system  mission  failure  rate  based  on  the  number  of 
failures  expected  when  the  software  is  used  in  its 
intended  environment. 

(2)  Another  aspect  of  reliability  is  an 
assessment  of  the  length  of  system  downtime  associated 
with  software  failures.  This  deals  with  the  time  it 
takes  to  restore  the  system  for  use  and  is  analogous  to 
hardware  mean  time  to  repair  (MTTR) . 

(3)  There  are  many  analytical  models  available 
to  estimate  software  reliability  as  a  function  of  test 
time.  These  reliability  models  have  the  benefit  of 
illustrating  the  improvement  in  software  reliability 
over  time  and  can  provide  estimates  of  total  test  time 
required,  based  on  a  failure  rate,  before  a  reliability 
objective  is  met.  Successive  estimates  of  the  failure 
rate  are  normally  obtained  from  the  cumulative 
probability  distribution  of  the  failures.  One  of  the 
principal  reasons  no  single  model  is  endorsed  here  is 
due  to  the  lack  of  consensus  regarding  the  particular 
pattern  or  distribution  associated  with  the  software 
failures  of  a  particular  system.  Although  no  single 
reliability  model  can  be  trusted  to  faithfully 
represent  future  software  performance  in  all 
circumstances,  the  judicious  selection  of  a  mo(l|:4gWj^ch 
closely  approximates  the  actual  behavior  of  the^^n  as' 
software  can  significantly  aid  in  the  estimation  of« 
future  reliability  performance. 

b.  Life  cycle  application.  See  paragraph  17-15b 
for  collecting  fault  profile  data.  Begin  collecting 
the  remainder  of  reliability  metric  data  during  system 
level  testing. 

c.  Algorithm/graphical  display. 

(1)  The  selection  of  a  reliability  model  is 
based  on  several  factors:  the  length  of  the  reliability 
test,  how  a  software  failure  is  defined,  the  adequacy 
of  the  operational  profile,  as  well  as  the  assumptions 
underlying  the  use  of  each  model.  For  example,  the 
Nonhomogenous  Poisson  Process  (NHPP)  model,  is  suitable 
for  many  cases  of  reliability  growth  measurement,  but 
involves  the  acceptance  of  some  basic  assumptions. 
"Poisson"  refers  to  a  particular  probability 
distribution  of  the  software  failures,  while  "non- 
hombgenous"  indicates  that  the  characteristics  of  the 
Poisson  distribution  will  vary  over  time.  The  NHPP 
also  assumes  that  faults  at  a  given  level  of  severity 
contribute  about  equally  to  the  failure  rate,  therefore 
the  repair  of  each  fault  will  also  have  about  an  equal 
effect  on  the  failure  rate.  This  means  that  the  NKPP 
is  well  suited  for  a  testing  environment  in  which 
repairs  are  conducted  in  response  to  software  failures 
leading  to  a  steadily  decreasing  rate  of  software 
failures  as  exhibited  in  Figure  17-20. 

(2)  Not  all  operational  environments  are  suited 
for  the  NHPP.  Once  a  selected  analytical  model  is 
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Reliability  Projection  Using  NHPP  Model 


Pigura  17-20.  saapla  Raliabillty  Modal  Projactlon 


applied,  it  is  important  to  determine  how  closely  the 
past  predictions  from  a  particular  model  for  a 
particular  data  set  reflect  the  actual  behavior  for 
that  data  set.  Various  statistical  and  qualitative 
methods  can  be  employed  to  determine  the  degree  of 
commonality  between  the  two  data  sets.  If  it  is 
determined  that  the  selected  model  is  not  an  accurate 
reflection  of  the  actual  behavior  then  apply  other 
models  until  a  good  fit  is  achieved.  The  most 
comprehensive  collection  of  prediction  models  is 
available  within  the  Statistical  Modeling  and 
Estimation  of  Reliability  Functions  for  Software 
(SMERFS)  software  package  developed  by  the  Government, 
other  software  tools  are  also  provided  within  SMERFS  to 
compare  a  model's  prediction  of  the  behavior  of  a 
software  application  with  the  actual  behavior  of  that 
software.  Data  requirements  on  software  faults  used  as 
input  to  the  analytical  models  are  described  in  the 
fault  profiles  metric,  paragraph  17-15. 

(3)  Use  fault  profile  metrics  throughout 
development  to  observe  trends  in  the  type  of  faults 
occurring  and  their  rates  of  closure. 

(4)  Other  software  reliability  data  will  be 
reported  as  test  incident  reports  (TIRs)  which  are 
chargeable  to  software.  All  calculations  to  derive 
software  reliability  values  (point  estimates,  lower 
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confidence  bounds,  mean  times,  median  times,  and 
maximum  values)  is  performed  using  the  procedures  and 
formulas  which  have  been  established  for  calculating 
all  other  TIR  data  in  accordance  with  Part  One  of  DA 
Pamphlet  73-1. 

(5) -  After  turnover  to  the  Government  for  DT  and 
OT,  compute  the  following  items  only  when  the  software 
is  being  used  and  stressed  in  accordance  with  its 
Operational  Mode  Summary /Mission  Profile  (OMS/MP) . 

(a)  Determine  the  computed  point  estimate  of 
mean  time  between  mission  failures  caused  by  system 
hardware  or  software  as  measured  during  the  test  event. 

(b)  Calculate  the  80%  lower  confidence  bound 
Value  of  mean  time  between  mission  failures  caused  by 
system  hardware  or  software. 

(c)  Determine  the  computed  point  estimate  of 
mean  time  between  mission  failures  caused  by  software 
as  measured  during  the  test  event. 

(d)  Calculate  the  80%  lower  confidence  bound 
value  of  mean  time  between  mission  failures  caused  by 
software. 

(e)  Compute  the  mean  time  to  restore  the 
system  to  operational  condition  after  a  software-caused 
system  failure  has  occurred.  Note:  In  the  calculations 
of  restoration  times  for  this  metric,  the  time  to 
restore  should  include  the  time  it  ta)ces  to 
reinitialize  the  system  and  recover  lost  data. 

(f)  Compute  the  median  time  to  restore  the 
system  to  operational  condition  after  a  software-caused 
system  failure  has  occurred. 

(g)  Calculate  the  maximum  95th  percentile 
value  of  time  to  restore  the  system  to  operational 
condition  after  a  software-caused  system  failure  has 
occurred . 

(5)  Figure  17—21  shows  computed  point  estimates 
of  system  mean  time  between  mission  failures,  the 
required  specification  value  and  associated  80%  lower 
confidence  bound  plotted  over  two  test  events.  Check 
at  each  failure  if  the  trend  is  towards  the 
specification  value.  Do  not  proceed  to  OT  unless  the 
computed  MTBF  is  greater  than  the  lower  80%  confidence 
bound. 

(6)  Figure  17-22  shows  the  computed  mean,  median 
and  95th  percentile  times  to  restore  the  system  to 
operational  status.  They  are  plotted  against  their 
corresponding  specification  values.  The  desired  trend 
is  for  the  computed  values  over  time  to  be  less  than  or 
equal  to  their  specified  values. 

d.  Data  requirements. 

(1)  All  data  requirements  from  the  fault 
profiles  metric 

(2)  Measured  software  failure  rate 

(3)  Projected  software  failure  rate 

(4)  Software  failure  rate  objective 
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Reliability 


Hours 


Figiira  17-21.  Saapla  Naan  Tiaa  Batvaan  Nisaien  Failuras 
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Flgura  17-22.  Saapla  Naan  Tiaa  to  Raatora  Systaa 


(5)  All  reliability,  availability,  and 
maintainability  incident  data  from  test  events 
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(6)  Test  identification  (such  as  Reliability 
Growth  Test  (RGT) ,  Developmental  Test  (DT)  and 
Operational  Test  (OT) ) 

(7)  The  required  specification  value  of  system 
Mean  Time  Between  Mission  Failure  (MTBF) 

(8) -  The  required  specification  values  for  mean, 
median  and  maximum  95th  percentile  mean  time  to  restore 
the  system  to  operational  status. 

e.  Frequency  of  reporting. 

(1)  Fault  profile  information  -  see  paragraph 
17-l5e. 

(2)  Monthly  for  data  based  on  fault  profile 
information  and  reliability  modeling. 

(3)  Government  DT/OT  -  gather  data  in  real  time 
as  incidents  occur  and  report  monthly. 

f.  Use/ interpretation. 

(1)  Fault  profile  metrics  indicate  rates  at 
which  faults  are  being  reduced  and  thus  reliability 
increased.  One  also  needs  to  simultaneously  consider 
the  test  coverage  metrics.  The  fault  profile 
information,  however,  says  nothing  about  how  often  the 
faults  remaining  in  the  software  will  be  encountered  by 
the  user. 

(2)  While  many  arguments  can  be  made  that  mean 
time  between  failure  is  an  inappropriate  measure,  it  is 
more  than  adequate  for  estimating  how  often  one  can 
expect  the  software  to  "fail”  in  a  field  environment  as 
long  as  inputs  are  of  the  type  and  in  relative 
proportion  to  what  will  be  encountered  in  field  use  and 
modules  are  exercised  with  the  relative  frequency 
expected  in  a  tactical  environment.  Measuring  MTBF 
only  when  the  system  and  software  are  being  used  in 
accordance  with  the  OMS/MP  insures  the  above 
conditions. 

(3)  When  using  fault  profiles  as  a  reliability 
measure,  refer  to  the  use  and  interpretation  guide 
previously  given  for  fault  profiles. 

(4)  Using  the  NHPP  algorithm,  progress  can  be 
tracked  using  various  reliability  growth  techniques. 

In  cases  where  the  system  MTBF  is  not  meeting  its 
requirement,  the  overall  failure  rate  of  the  software 
can  be  analyzed  by  using  this  model  and  the 
contribution  of  software  to  the  system  failure  rate  can 
be  investigated  by  other  means. 

g.  Rules  of  thumb.  No  formal  thresholds  are  given. 
However,  the  Government  should  not  accept  the  software 
until  the  fault  profile  open  anomaly  curve  flattens 
out,  closure  of  anomalies  approaches  the  open 
anomalies,  and  a  high  degree  of  breadth  and  depth  of 
testing  have  been  demonstrated.  Do  not  proceed  to  OT 
until  system  level  MTBF  (inclusive  of  both  hardware  and 
software)  have  been  demonstrated  with  high  confidence 
in  DT. 
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h.  References.  References  (k) ,  (m)  and  (u)  of 
Section  III  are  applicable  to  this  metric. 

Section  II 
Optional  Metrics 

17-17.  Manpower  metric 

a.  Purpose/description.  This  measure  provides  an 
indication  of  the  developer's  application  of  human 
resources  to  the  development  program  and  ability  to 
maintain  sufficient  staffing  to  complete  the  project. 

It  can  also  provide  indications  of  possible  problems 
with  meeting  schedule  and  budget.  It  is  used  to 
examine  the  various  elements  involved  in  staffing  a 
software  project.  These  elements  include  the  planned 
level  of  effort,  the  actual  level  of  effort,  and  the 
losses  in  the  software  staff  measured  per  labor 
category.  Planned  manpower  profiles  can  be  derived 
from  the  appropriate  planning  documents  submitted  to 
the  Government.  These  are  usually  provided  in  the 
contractor's  proposal  or  the  software  development  plan. 
The  planned  level  of  effort  is  the  number  of  labor 
hours  scheduled  to  be  worked  on  a  CSCI  each  month.  The 
planned  levels  are  compared  with  the  actual  levels  over 
a  given  time  period.  Deviations  between  planned  and 
actual  levels  are  monitored  to  ensure  that  the 
developer  is  meeting  the  necessary  staffing  criteria. 

b.  Life  cycle  application.  Track  for  entire  length 
of  development  (including  PDSS) . 

c.  Algorithm/graphical  display.  The  sample  graph 
shown  in  Figure  17-23  depicts  the  manpower  metric  for 
an  entire  system  over  all  labor  categories.  Figure  17- 
24  is  an  example  of  a  staffing  profile. 

d.  Data  requirements. 

(1)  Software  planning  documents  reflecting 
expected  staffing  profiles  and  labor  hour  allocations. 

(2)  For  each  labor  category  tracked: 

(a)  Labor  category  name 

(b)  For  each  experience  level  (experienced, 
special,  total) : 

-  number  of  personnel  planned  to  be  on 
staff  for  the  next  reporting  period 

-  number  of  personnel  actually  on  staff  in 
the  current  reporting  period 

-  number  of  unplanned  losses  in  personnel 
that  occurred 

-  number  of  labor  hours  that  are  planned 
to  be  expended  in  the  next  reporting 
period  (cumulative) 

-  number  of  labor  hours  actually  expended 
in  the  current  reporting  period  (cumula¬ 
tive) 

e.  Frequency  of  reporting.  Monthly. 
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f.  Use/ interpretation. 

(1)  Special  skills  personnel  are  those 
individuals  who  possess  specialized  software-related 
abilities  defined  as  crucial  to  the  success  of  the 
P^^'ticular  system.  For  example,  Ada  programmers  or 
artificial-  intelligence  experts  might  be  considered 
special  skills  personnel  for  one  project,  but  not 
necessarily  for  another  project. 

(2)  Experienced  personnel  are  defined  as  those 
individuals  with  a  minimum  of  three  years  experience  in 
software  development  for  similar  applications. 

(3)  Total  personnel  are  the  sum  of  experienced 
and  inexperienced  personnel.  Special  skills  personnel 
are  counted  within  the  broad  categories  of  experienced 
and  inexperienced,  but  are  also  tracked  separately. 

(4)  Tracking  by  individual  labor  category  can  be 
done  for  programs  with  large  staffs  or  to  monitor 
aspects  of  a  program  which  are  deemed  worthy  of  special 
attention.  For  many  projects,  software  quality 
assurance  people  might  be  tracked  as  a  separate 
category. 

(5)  The  software  staff  includes  only  those 
engineering  and  management  personnel  directly  involved 
with  software  system  planning,  requirements  definition, 
design,  coding,  integration,  test,  documentation, 
configuration  management,  and  quality  assurance. 

Losses  and  gains  for  each  category  specified  above 
should  be  tracked  monthly  to  indicate  potential  problem 
areas.  Personnel  who  have  been  replaced  are  still 
counted  as  a  loss.  High  turnover  of  experienced 
personnel  can  adversely  affect  project  success.  Also, 
for  example,  adding  many  personnel  (beyond  those 
numbers  planned)  late  in  the  development  process  may 
pi’ovide  an  indication  of  impending  problems.  Turnover 
of  key  people  must  also  be  watched  closely. 

(6)  Significant  deviations  from  planned  levels 
indicate  potential  problems  with  staffing  the  various 
software  development  activities.  Deviations  between 
actual  and  planned  levels  can  be  detected,  explained 
and  corrected  before  they  negatively  Impact  the 
development  schedule.  The  losses  in  the  staff  can  be 
monitored  to  detect  a  growing  trend  or  significant  loss 
of'  experienced  staff.  This  indicator  assists  the 
Government  in  determining  whether  the  developer  has 
scheduled  a  sufficient  number  of  employees  to  produce 
the  product  in  the  time  allotted. 

(7)  The  shape  of  the  staff  profile  trend  curve 
tends  to  start  at  a  moderate  level  at  the  beginning  of 
the  contract,  grow  through  design,  peak  at 
coding/testing  and  diminish  near  the  completion  of 
integration  testing.  Individual  labor  categories, 
however,  are  likely  to  peak  at  different  points  in  the 
life  cycle.  The  optimum  result  would  show  little 
deviation  between  the  planned  and  actual  levels  for 
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each  category  with  losses  kept  to  a  minimum.  Specific 
attention  should  be  paid  to  any  case  where  there  is  a 
significant  deviation  (plus  or  minus  10%)  between 
actual  and  planned  values.  If  the  actual  value  is  10% 
or  more  than  the  planned  value,  this  may  suggest  that 
the  developer: 

(a)  Underestimated  the  work  involved. 

(b)  Found  out  that  the  task  was  more 
complicated  than  expected. 

(c)  Did  not  perform  the  work  efficiently. 

(d)  Is  ahead  of  schedule. 

(e)  Is  behind  schedule  and  is  adding  manpower 
to  catch  up. 

(f)  Is  adding  people  to  make  up  for  a  lack  of 
experienced  ones. 

(8)  If  the  actual  levels  are  less  than  planned 
levels,  this  may  suggest  that  the  developer: 

(a)  Overestimated  the  work  involved. 

(b)  Did  not  perform  the  task  completely. 

(c)  Did  not  accurately  determine  the 
complexity  of  the  effort. 

(d)  Performed  the  work  efficiently. 

(e)  Did  not  assign  adequate  manpower  to  the 

task. 

(f)  Misinterpreted  the  task  or  requirements. 

(g)  Is  ahead  of  schedule. 

(9)  In  cases  of  large  deviation  the  developer 
should  be  required  to  determine  the  cause  and  report 
any  corrective  actions  necessary. 

(10)  The  manpower  metrics  are  used  primarily  for 
project  management  and  do  not  necessarily  have  a  direct 
relationship  with  other  technical  and  maturity  metrics. 
For  example,  manpower  levels  are  usually  higher  during 
testing  activities.  This  does  not  necessarily  reflect 
an  increase  in  the  quality  levels  of  the  product  or 
suggest  that  the  depth  of  testing  metrics  will  be 
higher. 

(11)  The  manpower  metrics  should  be  used  in 
conjunction  with  the  development  progress,  test 
coverage  and  cost  metrics. 

(12)  The  value  of  this  metric  is  somewhat  tied 
to  the  accuracy  of  the  developer's  development  and 
staffing  plan,  as  well  as  to  the  accuracy  of  the  labor 
reporting  system. 

g.  Rules  of  thumb. 

(1)  A  high  ratio  of  total  to  experienced 
personnel  is  undesirable.  A  ratio  of  3:1  is  typical. 

(2)  Significant  deviations  from  the  planned 
staffing  profile,  as  well  as  a  high  turnover  rate  in 
any  category,  should  be  investigated  so  as  to  minimize 
risk  to  the  Government. 

(3)  When  80%  of  the  planned  or  budgeted 
resources  have  been  expended,  that  fact  should  be 
highlighted  to  the  Government  and  documented  in  the 
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monthly  reports.  Closer  attention  should  then  be  paid 
to  the  remaining  resources. 

h.  References.  References  (a) ,  (b) ,  (g)  and  (s)  of 
Section  III  are  applicable  to  this  metric. 

17-18.  Davalopaant  progress  metric 

a.  Purpose /description.  The  development  progress 
metrics  provide  indications  of  the  degree  of 
completeness  of  the  software  development  effort,  and 
can  be  used  to  judge  readiness  to  proceed  to  the  next 
stage  of  software  development. 

b.  Life  cycle  application.  Begin  collecting  at  SRR 
and  continue  for  the  entire  software  development  phase. 

c.  Algorithm/graphical  display. 

(1)  A  sample  of  the  development  progress  metric 
is  shown  in  Figure  17-25. 


DEVELOPMENT  PROGRESS 


%  C8U« 


Program  Month 


■  Figure  17-25.  Sample  Development  Progress  Metric 

(2)  The  following  calculations  can  be  performed 
at  either  the  CSC,  CSCI,  or  system  level. 

(a)  Compute  percent  of  CSUs  100%  designed. 

(b)  Compute  percent  of  CSUs  100%  coded  and 
successfully  unit  tested. 

(c)  Compute  percent  of  CSUs  100%  integrated. 

(3)  Additionally,  using  the  requirements 
traceability  matrix,  one  can  plot  functionality  (which 
has  been  developed  and  verified)  versus  time  as  a 
measure  of  development  progress. 
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d.  Data  requirements. 

(1)  CSCI/CSC/CSU  development,  test  and 
integration  schedules 

(2)  For  each  CSCI: 

(a)  Number  of  CSUs  per  CSCI 

Cb)  Number  of  CSUs  100%  designed  and  reviewed 
by  the  Government  (cumulative) 

(c)  Number  of  CSUs  100%  coded  and 
successfully  unit  tested  (cumulative) 

(d)  Number  of  CSUs  100%  integrated  into  a  CSC 
or  CSCI  (cumulative) 

(e)  Nvimber  of  CSUs  planned  to  be  100% 
designed  and  reviewed  by  the  Government  for  the  next 
reporting  period  (cumulative) 

(f)  Number  of  CSUs  planned  to  be  100%  coded 
and  successfully  unit  tested  for  the  next  reporting 
period  (cumulative) 

(g)  Number  of  CSUs  planned  to  be  100% 
integrated  into  a  CSC  or  CSCI  for  the  next  reporting 
period 

e.  Frequency  of  reporting.  Monthly.  ^-^on  of 

f.  Use/interpretation. 

(1)  "Successfully”  tested  is  defined  as 
completing  all  test  cases  (required  test  covera- 
depth)  with  no  defects.  "Integrated”  is  define  . 
being  actually  and  logically  connected  (in  a  static- 
sense)  with  all  required  modules.  (Dynamic  tasking  is 
not  considered  here) . 

(2)  Design,  coding,  unit  testing,  and 
integration  of  CSUs  should  progress  at  a  reasonable 
rate.  Plotting  the  progress  in  these  three  categories 
versus  what  was  originally  planned  can  indicate 
potential  problems  with  schedule  and  cost.  In  certain 
instances,  consideration  must  be  given  to  a  possible 
re-baselining  of  the  software  (e.g.,  in  an  evolutionary 
approach)  or  if  one  simply  must  add  modules  due  to 
changes  in  the  requirements. 

p)  The  development  progress  metrics  should  be 
used  with  the  test  coverage  metrics  (breadth  and  depth 
of  testing)  to  assess  the  readiness  to  proceed  itctJi 
formal  Government  test.  They  should  also  be  use*  with 
the  requirements  traceability  metrics  so  that  progress 
can  be  tracked  in  consonance  with  the  tracing  of 
requirements.  Additionally,  it  can  be  used  with  the 
schedule  metric  to  help  evaluate  schedule  risk. 

(4)  The  development  progress  metrics  should  be 
used  with  the  manpower  metrics  to  identify  areas  where 
the  developer  is  experiencing  problems.  Also,  using 
these  metrics  with  the  computer  resource  utilization 
metrics  can  ensure  that  the  actual  utilization  is 
representative  of  a  complete  system.  Finally,  special 
attention  should  be  given  to  the  development  progress 
of  high  complexity  CSUs. 
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(5)  These  metrics  pass  no  judgement  on  whether 
the  objectives  in  the  development  plan  can  be  achieved. 

g.  Rules  of  thumb. 

(1)  One  hundred  percent  (100%)  of  all  CSUs 
should  be  designed  prior  to  proceeding  beyond  CDR  for 
the  appropriate  CSCI. 

(2)  One  hundred  percent  (100%)  of  all  CSUs 
should  be  coded,  successfully  tested,  and  integrated 
before  proceeding  to  a  formal  system  level  Government 
test. 

h.  References.  References  (a)  and  (g)  of  Section 
III  are  applicable  to  this  metric. 

section  ZIZ 
Metrics  Rafaraacas 

17-19.  Rafarancas 

Additional  detail  and  related  materials  on  the  metrics 
described  in  this  chapter  can  be  found  in  these 
publications. 

a.  AFSCP  800-43,  Software  Management  Indicators, 

Air  Force  Systems  Command,  31  January  1986. 

b.  AMC-P  70-13,  Software  Management  Indicators, 
Management  Insight,  31  January  1987. 

c.  AMC-P  70-14,  Software  Quality  Indicators,  20 
January  1987 . 

d.  ARPAD-TR-88005,  The  Complexity  Analysis  Tool, 
October  1988. 

e.  CMU/SEI-87-TR-23 ,  A  Method  for  Assessing  the 
Software  Engineering  Capability  of  Contractors, 
Carnegie-Mellon  University  Software  Engineering 
Institute  Technical  Report,  September  1987. 

f.  DA  Pam  XX-XX,  Operational  Requirements  for 
Automated  Capabilities  (Draft),  2  January  1992. 

g.  ESD-TR-85-145,  Software  Reporting  Metrics,  The 
Mitre  Corporation,  MTR  9650  Revision  2,  November  1985. 

h.  IEEE  Standard  982.1-1988,  IEEE  Standard 
Dictionary  of  Measures  to  Produce  Reliable  Software,  30 
April  1989. 

i.  IEEE  Standard  982.2-1988,  IEEE  Guide  for  the  Use 
of  IEEE  Standard  Dictionary  of  Measures  to  Produce 
Reliable  Software,  12  June  1989. 

•  j.  MIL-HDBK-WBS.SW,  Wor)c  Breakdown  Structure  for 
Software  Cost  Reporting  (Draft),  l  October  1991. 

k.  NAVSWC  TR  84-373,  Statistical  Modeling  and 
Estimation  of  Reliability  Functions  for  Software 
(SMERFS)  Users  Guide  Revision  2,  March  1991. 

l.  NBS  500-99,  Structured  Testing:  A  Software 
Testing  Methodology  Using  the  Cyclomatic  Complexity 
Metric,  National  Bureau  of  Standards,  December  1982. 

m.  RADC-TR-82-263,  Software  Reliability  Modelling 
and  Estimation  Techniques,  Rome  Air  Development  Center, 
October  1982. 
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n.  Beizer,  Boris,  Software  System  Testing  and 
Quality  Assurance,  Van  Nostrand  Reinhold,  1984. 

o.  Buckley,  Fletcher  J. ,  Standard  Set  of  Useful 
Software  Metrics  Is  Urgently  Needed,  Computer,  July 
1989. 

p.  Craig,  Measuring  Effectiveness  and  Adequacy  of 
System  Testing,  Conference  Proceedings,  Software  Test 
and  Validation,  National  Institute  for  Software  Quality 
and  Productivity,  Inc.,  September  1987. 

q.  Design  Complexity  Metrics,  T.J.  McCabe  & 
Associates,  Inc.,  May  1988. 

r.  Draft  Guide  for  the  Use  of  Standard  Dictionary  of 
Measures  to  Produce  Reliable  Software,  IEEE  Computer 
Society,  May  1988. 

s.  Revised  Implementation  Guidelines  for  Software 
Management  and  Quality  Indicators  for  AFATDS,  30  July 
1989. 

t.  Software  "Engineering  Institute  Quality  Subgroup 
Working  Papers,  untitled,  November  1989. 

u.  Software  Measurement  Models,  Data  &  Analysis 
Center  for  Software,  July  1987. 

V.  Walsh,  A  Software  Reliability  Study  Using  a 
Complexity  Measure,  1982. 


17-60 


30  8«pt«fflbttr  1992 


DA  Pam  73-1,  Part  Savan  (Draft) 


Appandiz  A 
Saetion  I 

Raquirad  Publications 

AR  73-1,  Test  and  Evaluation  Policy,  dated  6  DEC  90. 
Cited  throughout  this  part  of  DA  Pamphlet  73-1.  Cited 
on  pages  1-1,  l-4,  1-5,  1-6,  1-8,  1-11,  2-1,  3-10,  3- 
12,  3-14,  5-1,  5-5,  5-11,  8-1,  9-3,  11-3,  11-4,  11-6, 
12-2,  14-1,  15-1,  15-4,  and  17-1. 

AR  380-19,  Information  Systems  Security,  dated  4  SEP 
90.  cited  on  page  3-8. 

DA  Pamphlet  73-1,  Part  One,  Test  and  Evaluation 
Procedures  Guide.  Cited  on  page  17-50. 

DA  Pamphlet  73-1,  Part  Two,  Test  and  Evaluation  Master 
Plan  (TEMP)  Format,  Review  and  Approval  Procedures. 
Cited  on  pages  4-10  and  5-5. 

DA  Pamphlet  73-1,  Part  Three,  Critical  Operational 
Issues  and  Criteria  (COIC)  Development,  Review  and 
Approval  Guidelines.  Cited  on  page  5-4. 

DA  Pamphlet  73-1,  Part  Four,  Developmental  Test  and 
Evaluation  (DTiE)  Guidelines.  Cited  on  pages  5-6,  5-7, 
8-1  and  14-1. 

DA  Pamphlet  73-1,  Part  Five,  Operational  Test  and 
Evaluation  (OT&E)  Guidelines.  Cited  on  Pages  5-6,  5- 
10,  5-11,  5-12,  9-3,  9-4,  and  15-2. 


Section  iz 

Related  Publications 

AFOTECP  800.2,  Volume  4,  Software  Usability  Evaluator's 
Guide. 

< 

AR  15-38,  Test  Schedule  and  Review  Committee. 

AR  -25-1,  The  Army  Information  Resources  Management 
Program. 

AR  25-3,  Army  Life  Cycle  Management  of  Information 
Systems . 

AR  25-5,  Information  Management  with  a  Sustained  Base. 

AR  40—60,  Policies  and  Procedures  for  the  Acquisition 
of  Medical  Materiel. 

AR  70-1,  Systems  Acquisition  Policy  and  Procedures. 


A-1 

» p'.  ■*  • 


30  Soptcabor  1992 


DA  Pun  73-1,  Part  Savan  (Draft) 


AR  70-15,  Materiel  Change  Management. 

AR  70-25,  Use  of  Volunteers  as  Subjects  of  Research. 

AR  70-37,  Configuration  Management. 

AR  71-9,  Materiel  Objectives  and  Requirements. 

AR  105-7,  Army  Training  and  Audio-Visual  Support. 

AR  350-38,  Training  Device  Policies  and  Management. 

AR  525-1,  Strategic  Systems. 

AR  700-142,  Materiel  Release. 

CMU/SEI-87-TR-23 ,  A  Method  for  Assessing  the  Software 
Engineering  Capability  of  Contractors,  Carnegie  Mellon 
University  Software  Engineering  Institute  Technical 
Report . 

DA  Pamphlet  25-6,  Configuration  Management  for 
Automated  Information  Systems. 

DA  Pamphlet  700-142,  Materiel  Release. 

DA  Pamphlet  XX-XX,  Operational  Requirements  for 
Automated  Capabilities  (Draft) . 

DI-A-7089,  Conference  Minutes. 

DI-MCCR-80014A,  Software  Test  Plan. 

DI-MCCR-80015A,  Software  Test  Description. 

DI-MCCR-80017A,  Software  Test  Report. 

DI-QCIC-80572 ,  Software  Quality  Program  Plan. 

DI-S-30559,  Technical  Operating  Report  (TOR). 

D0D-STD-1467A,  Post-Deployment  Software  Support. 

DOD-STD-1838,  Common  Ada  Programming  Support 
Environment  Interface  Set  (CAIS) . 

DOD-STD-2167A,  Defense  Systems  Software  Development. 

DOD-STD-2168 ,  Defense  System  Software  Quality  Program. 

DOD-STD-7935A,  DOD  Automated  Information  System  (AIS) 
Documentation  Standards. 

DODD  3405.1,  Computer  Programming  Language  Policy. 

A-2 


n- 


30  8«pt«ab«r  1992  DA  Pan  73-1,  Part  Savan  (Draft) 


DODD  3405.2,  Use  of  Ada  in  Weapons  Systems. 

DODD  5000.1,  Defense  Acquisition. 

DODD  7920.1,  Life-Cycle  Management  of  Automated 
Information  Systems  (AISs) . 

DODI  5000.2,  Defense  Acquisition  Management  Policies 
and  Procedures. 

DODI  7000.2,  Performance  Measurement  for  Selected 
Acquisitions. 

DODI  7920.2,  Automated  Information  Systems  (AIS)  Life- 
Cycle  Management  Review  and  Milestone  Approval 
Procedures . 

DODI  7920.4,  Baselining  of  Automated  Information 
Systems  (AISs) . 

MIL-HDBK-245B,  Writing  Statements  of  Work  (SOWs) . 

MIL-HDBK-WBS . SW ,  Work  Breakdown  Structure  for  Software 
Elements  (Second  Draft) . 

MIL-STD-480B,  Configuration  Control  -  Engineering 
Changes,  Deviations  and  Waivers. 

MIL-STD-490A,  Specification  Practices. 

MIL— STD— 8 8 IB ,  Work  Breakdown  Structures  for  Defense 
Materiel  Systems. 

MIL-STD-1521B,  Technical  Reviews  and  Audits  for 
Systems,  Equipments,  and  Computer  Software. 

MIL-STD-1815A,  Ada  Programming  Language. 

NBS  500-99,  Computer  Science  and  Technology: 

Structured  Testing:  A  Software  Testing  Methodology 
Using  Cyclomatic  Complexity  Metric. 

0MB  Circular  A,  Purchases  of  ADP  Equipment. 

RADC-TR-87-171,  Volumes  1  and  2,  Methodology  for 
Software  Reliability  Prediction. 


A-3 


30  8*ptuib«r  1992  DA  Pab  73-1,  Part  Savan  (Draft) 


Appandix  B 

BtataBant  of  Work  (BOW) 

B-1.  Btataaant  of  Work  daaeription 

a.  Software  test  and  evaluation  is  an  emerging 
technolo^  for  which  specific  requirements  are  not 
well-defined  in  existing  military  specifications  and 
standards.  Specific  tasks  for  planning  and  executing 
software  T&E  must  be  tailored  to  the  technical  and 
management  characteristics  of  each  software 
development.  The  statement  of  work  (SOW)  defines  the 
work  tasks  and  services  which  must  be  performed  in  the 
T&E  program  for  software  development. 

b.  MIL-HDBK-245B,  Notice  1  of  31  December  1989 
provides  formal  requirements  for  developing  and 
implementing  the  SOW.  The  SOW  provides  all  that 
information  which  cannot  be  defined  in  the  limited 
scope  of  technical  specifications  and  contract  data 
requirements  lists  (CDRLs) .  Specifications  are  limited 
to  descriptions  of  technical  and  performance 
requirements  of  the  software  products.  CDRL  items  are 
limited  to  describing  data  to  be  submitted  during  the 
software  development.  Examples  of  information  which 
can  be  presented  in  an  SOW  for  software  T&E  are: 
background  information,  program  management  objectives, 
software  technical  objectives,  and  software  functional 
needs. 

c.  The  SOW  is  part  of  a  binding  legal  document,  a 
contract,  and  must  be  prepared  carefully  and 
accurately.  Errors  and  shortcomings  can  be  costly  to 
correct. 

B-2.  Statement  of  Work  criteria 

a.  In  order  to  impact  the  specific  software 
development  tasks,  software  T&E  personnel  must  be 
involved  in  developing  the  SOW.  Specific  items  which 
should  be  addressed  by  software  T&E  personnel  in  SOW 
development  are: 

(1)  Distribution  of  CDRL  items  on  a  DD  Form  1423 
should  include  independent  evaluators,  the  PM's 
software  matrix  support  activity,  PDSS-LCSEC/CDA 
personnel ; 

(2)  Software  tests  must  permit  derivation  of 
data  which  support  stated  software  maturity 
measurements  and  the  selected  metrics  set; 

(3)  Contracted  software  testing  must  provide 
usable  data  for  Government  evaluations. 

b.  Some  specific  software  T&E  issues  which  should 
be  addressed  in  all  SOWs  for  software  development  are 
provided  in  checklist  form  in  Figure  B-l.  Checklist 
items  with  a  subjective  score  of  four  or  less  should  be 
clarified  or  elaborated. 


B-l 


n-<»H 


30  8«pt«mb«r  1992 


OA  Paa  73-1,  Part  8avan  (Draft) 


B'3.  8aBpla  80W  paragraphs 

A  selection  of  sample  software  related  SOW  paragraphs 
is  provided  for  reference.  These  may  be  tailored 
individually,  or  as  logical  groups  dependent  on 
acquisition  needs.  Some  paragraphs  require  system- 
specific  values  to  be  inserted.  Paragraphs  dealing 
with  different  aspects  of  the  same  area,  such  as  growth 
capacity  and  stress  testing,  should  be  coordinated  to 
avoid  duplicating  or  conflicting  requirements. 

a.  Software  Reviews  and  Audits:  The  contractor 
shall  host  system  and  software  reviews  and  audits  lAW 
MIL-STD-1521B  (i.e. ,  SRR,  SDR,  SSRs,  PDRs,  CDRs,  TORs, 
FCA/PCA) ,  as  applicable.  Reviews  shall  be  iterative 
and  conducted  at  action  officer  levels.  PDRs  and  CDRs 
shall  provide  demonstrations  for  user  review  during 
these  events.  Software  and  docximentation  necessary  to 
support  these  reviews  shall  be  submitted  to  the 
Government  in  a  timely  manner,  so  as  to  allow  for 
Government  review  and  preparation.  Reviews  shall  be 
held  as  iterative  working  group  efforts  among 
Government  action  officers  and  contractor  personnel. 
Results  of  formal  reviews  and  audits  shall  be  reported 
in  accordance  with  DI-A-7089. 

b.  Software  Quality  Program:  The  contractor  shall 
develop  and  implement  a  software  quality  program  lAW 
DOD-STD-2168  to  complement  the  software  development. 

The  software  quality  program  shall  be  documented  lAW 
DI-QCIC-80572  and  submitted  to  the  Government  for 
approval. 

c.  Software  Quality  Indicators;  The  contractor 
shall  establish  procedures  and  methodologies  to  gather 
and  evaluate  data  to  implement  software  quality  metrics 
based  upon  the  quality  indicators  defined  in  DA 
Pamphlet  73-1,  Part  Seven,  Chapter  17.  The  metrics 
shall  be  tailored  to  the  software  development.  The 
contractor  shall  describe  the  methodologies  and 
procedures  used  to  implement  the  software  quality 
metrics  in  the  software  development  plan.  Metrics 
reports  shall  be  prepared  lAW  (insert  the  appropriate 
DIDs  from  Appendix  C) . 

d.  Software  Management  Indicators:  The  contractor 
shall  establish  procedures  and  methodologies  to  gather 
and  evaluate  data  to  implement  the  cost,  schedule  and 
computer  resource  utilization  software  management 
metrics  based  upon  management  indicators  defined  in  DA 
Pamphlet  73-1,  Part  Seven,  Chapter  17.  The  metrics 
shall  be  tailored  to  the  software  development.  The 
contractor  shall  describe  the  methodologies  and 
procedures  used  to  implement  the  software  management 
metrics  in  the  software  development  plan.  Metrics 
reports  shall  be  prepared  lAW  (insert  the  appropriate 
DIDs  from  Appendix  C) . 

e.  Software  Requirements  Metrics:  The  contractor 
shall  establish  procedures  and  methodologies  to  gather 
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and  evaluate  data  to  implenent  software  requirements 
metrics  (requirements  traceability  and  requirements 
stability)  based  on  metrics  indicators  defined  in  DA 
Pamphlet  73-1,  Part  Seven,  Chapter  17.  The  metrics 
shall  be  tailored  to  the  software  development.  The 
contractor'  shall  describe  the  methods  and  procedures 
used  to  implement  the  software  requirements  metrics  in 
the  software  development  plan.  Metrics  reports  shall 
be  prepared  lAW  (insert  the  appropriate  DIDs  from 
Appendix  C) . 

f.  Software  Security:  The  security  level  for  the 

software  is  _  (insert  applicable  security  level) , 

as  defined  in  ^  380-19.  The  contractor  shall  identify 
applicable  design  requirements  to  meet  the  security 
level  for  this  software  development,  including  the 
physical  security  classification  requirements  for 
documentation,  software  security  provisions  for  any 
classified  computer  interface,  and  access  to  and 
storage  of  classified  computer  information. 

g.  Quantification  of  Software  Reliability:  The 
contractor  shall  identify  all  software/code  related 
anomalies,  and  determine  if  they  fall  under  any  of  the 
following  categories: 

(1)  Operational  Mission  Failures  (OMF)  (Priority 
1  and  2)  -  those  which  would  have  caused  an  OMF 
in  the  fielded  system. 

(2)  Non-OMF  (Priority  3)  -  those  which  would 
have  caused  a  significant  degradation  of  mission 
functionality  in  the  fielded  system  but  not  an 
OMF. 

These  two  categories  of  anomalies  shall  Ise  separately 
documented  and  reported  to  the  Government,  on  a  monthly 
basis,  with  a  rationale  for  the  selection  of  the 
category  for  each  anomaly.  In  addition,  each  OMF  and 
non-OMF  anomaly  shall  be  reported  documenting  the 
frequency  of  occurrence  in  units,  based  on  application 
(i.e.,  cycles  of  ballistic  computations,  time  for 
communications  processing,  miles  for  navigation 
processing,  etc.).  This  data  will  be  analyzed  by  the 
Government  in  generation  of  software  reliability. 

h.  Independent  Verification  and  Validation  (IV6V) : 
The.  Government  reserves  the  right  to  perform,  or  have  a 
designated  representative  perform,  IVfiV  of  the  software 
being  developed  under  this  effort  at  any  time  during 
the  contract  period.  The  contractor  shall  ensure 
Government/ IVSV  agent  interface  and  data  exchange  in 
support  of  any  IVSV  activity.  To  accomplish  this,  the 
contractor  agrees  to  provide  access  to  any  and  all 
facilities,  security  clearance,  data,  documentation, 
plans,  software  development  folders/files,  and 
applicable  software  code  necessary  to  satisfy  the 
purpose  and  objectives  of  the  IVSV.  The  Government 
will  notify  the  contractor  at  least  five  (5)  worlcing 
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days  prior  to  any  IV&V  activity  and  will  state  the 
purpose  and  objectives  of  the  IV&V  activity. 

i.  Software  Cyclomatic  Complexity:  The  contractor 
shall  comply  with  the  following  software  complexity 
requirements  lAW  NBS  500-99.  The  software  cyclomatic 
complexity^  shall  not  exceed  a  complexity  limit  of  seven 
for  the  PDL  and  nine  for  source  code  derived  from  the 
PDL.  A  software  unit,  modified  by  corrective  action  in 
satisfaction  of  any  problem/ change  report,  software 
trouble  report  (STR),  or  other  record,  shall  not  exceed 
a  complexity  limit  of  twelve.  The  contractor  shall 
submit  a  summary  of  cyclomatic  complexity  with  the 
rationale  to  the  Government  for  comment  and  approval  at 
the  program  reviews,  as  specified  herein.  The 
complexity  shall  be  computed  and  documented  along  with 
the  documentation  of  PDL,  code  and  software  change 
requests,  and  shall  include  statements  detailing  the 
rationale  for  any  modules  exceeding  the  above 
complexity  limits.  The  cyclomatic  complexity  metric 
shall  be  applied  during  unit  and/or  module  level 
testing,  as  described  in  NBS  500-99.  Identif icativ 
paths  and  their  corresponding  test  case/drivers  shall 
be  documented  in  the  software  development  folders 
(SDFs)  or  program  folders  (PFs) . 

j.  Software  Tools:  The  contractor  shall  identify 
all  CASE  tools  used  during  the  software  development 
effort  and  document  them  in  the  software  development 
plan  (SDP) .  CASE  tools  must  be  used  during  the  Ada 
software  development  process.  These  CASE  tools  shall 
support  the  design,  coding,  documentation, 
configuration  management  and  unit/ system  test  needed 
for  CSCI/HWCI  development  as  specified  in  OOD-STD- 
2167A.  The  contractor  shall  identify  any  other 
software  tools  that  are  required  and  shall  maximize 
compatibility  with  DOD-STD-1838,  "Common  Ada 
Programming  Support  Environment  Interface  Set  (CAIS)." 
The  contractor  shall  also  use  DOD-STD-1467A  as  a  guide 
for  these  efforts. 

k.  Test  Hooks:  The  contractor  shall  incorporate 
test  hooks  (software  test  points)  into  the  operational 
code.  These  test  hooks  are  required  to  permit  the 
diagnosis  of  hardware,  software  and  operator  induced 
faults,  and  to  evaluate  hardware/ software  performance 
by  providing  the  capability  to  monitor  the  process  and 
flow  within  the  software.  The  contractor  shall 
identify  the  interfaces  and  monitoring/recording 
provisions  (hardware  and  software)  that  are  designed  in 
the  embedded  operational  code,  which  allow  the  use  of 
test  software  in  the  evaluation  of  the  operational 
system,  and  shall  document  these  in  an  engineering 
report  in  contractor  format.  The  test  hardware  and 
software  used  to  access  these  test  hooks  for  the 
evaluation  of  the  operational  system  shall  be 
documented  in  this  report. 


B-4 


n- 


30  8«ptwnb«r  1992 


DA  Pan  73-1 »  Part  Savan  (Draft) 


l.  Software  Testing:  The  contractor  shall  develop, 
implement,  and  maintain  a  comprehensive  software  test 
program  lAW  DOD-STD-2167A.  The  contractor 
responsibility  includes  all  information  required  for 
CSCI-level  testing,  level  A,  level  B,  and  formal 
qualification  testing  (FQT) .  These  tests  shall  be 
completed  prior  to  Government  Developmental  Test.  This 
information  shall  be  documented  in  the  software  test 
plan  (STP)  (DI-MCCR-80014A)  and  the  software  test 
description  (STD)  (DI-MCCR-80015A) .  Both  documents 
shall  be  delivered  to  the  Government  for  approval. 

m.  Unit  Level  Testing:  The  contractor  shall 
perform  testing  at  the  unit  level,  including  regression 
testing.  Units  shall  be  tested  prior  to  integration 
with  other  software.  Unit  level  test  information, 
including  identification  of  test  paths  and  their 
corresponding  test  cases/drivers  and  test  results, 
shall  be  maintained  in  the  software  development  folders 
(SDFs) .  The  contractor  shall  regression  test  all  code 
utilizing  paths  generated  from  the  cyclomatic 
complexity  method  defined  in  NBS  500-99.  The  paths 
chosen  shall  completely  exercise  all  new/modified  code 
and  requirements.  The , Government  shall  have  complete 
access  to  the  SDFs. 

n.  CSC  Level  Testing:  The  contractor  shall  perform 
testing  at  the  CSC  integration  level,  including 
regression  testing.  The  contractor  shall  use  unit /CSC 
interrelationship  diagrams  or  compiler-generated  cross- 
reference  listings  and  perform  regression  tests 
accordingly.  The  contractor  shall  test  all  units 
subordinate  to  the  modified  unit  to  assure  that  a 
change  did  not  result  in  performance  degradation.  CSC- 
level  test  information,  including  identification  of 
test  paths  and  their  corresponding  test  cases/drivers 
and  test  results,  shall  be  maintained  in  the  SDFs.  The 
contractor  shall  regression  test  all  code  utilizing 
paths  generated  from  the  cyclomatic  complexity  method 
defined  in  NBS  500-99.  The  paths  chosen  shall 
completely  exercise  all  new/modified  code  and 
requirements.  The  Government  shall  have  complete 
access  to  the  SDFs. 

o.  Formal  Qualification  Test  (FQT) :  The  contractor 
shall  conduct  FQT  lAW  the  approved  STP  and  STDs.  All 
FQT  shall  be  successfully  completed,  i.e.,  100%  of  all 
tests  conducted  shall  have  passed  prior  to  accepting 
the  software  for  Government  test  and  evaluation,  prior 
to  Developmental  Test  and  Operational  Test.  The 
contractor  shall  conduct  FQTs  as  follows: 

(1)  Level  A:  Bench  level  on  each  CSCI 

(2)  Level  B:  Bench  level  with  either  actual  or 
emulated  associated  subsystem (s) 

p.  Stress  Testing:  The  contractor  shall  conduct 
stress  testing  at  both  levels  A  and  B  as  specified 
below.  Stress  testing  shall  be  performed  immediately 
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subsequent  to  each  level  FQT.  Contractor-performed 
stress  testing  shall  be  conducted  and  the  results 
reported  to  the  Government  in  contractor  format. 

Stress  testing  shall  include  the  following: 

(1)  Functional  (all  levels): 

The  software  shall  be  tested  up  to  and  beyond  its 
designed  capacities.  The  contractor  shall  fully 
demonstrate  the  system's  ability  to  perform  within  the 
limits  of  its  design  capacities,  and  demonstrate  the 
ability  to  degrade  safely  during  periods  of  operation 
beyond  the  design  limits. 

(2)  Duration  (level  B  only): 

The  operational  software  shall  be  tested  for  a  period 
of  24  continuous  hours  under  loading  conditions  and 
other  conditions  representing  150%  of  the  system's 
OMS/MP. 

(3)  Throughput 

(a)  CPU  Loading  (all  levels) :  The 
contractor  shall  define  peak  operational  load  and 
demonstrate  that  the  software  shall  not  degrade  under 
twice  the  load. 

(b)  Database  Loading  (Level  A  only) :  The 
contractor  shall  define  normal  database  loading  and 
demonstrate  that  the  software  shall  not  degrade  when 
the  data  traffic  is  doubled. 

(c)  Memory  RAM  (Level  A  only):  The 
contractor  shall  demonstrate  that  the  software  shall 
not  degrade  during  peak  CPU  loading  (a.  above)  when  50% 
of  RAM  (Random  Access  Memory)  is  made  unavailable  to 
the  software  program. 

q.  Errors  During  Testing;  Any  error  during  FQT 
shall  constitute  a  failed  test,  and  shall  conditionally 
negate  a  successful  FQT  conducted  at  a  lower  level,  as 
applicable.  A  regression  analysis  (using  NBS  500-99 
unit/CSC  interrelationship  diagrams,  compiler  generated 
cross  reference  listings,  etc.)  shall  be  performed  as 
part  of  the  corrective  action  process.  The  software 
shall  be  regression  tested,  as  necessary.  The 
Government  must  approve  the  regression  test  procedures 
prior  to  the  start  of  test.  The  Government  reserves 
the  right  to  witness  all  testing.  No  software  release 
will  be  accepted  for  Government  use  without  Government 
approval. 

(1)  Unit  and  CSC  Level;  The  contractor  shall 
perform  regression  testing  at  the  unit  and  CSC  level. 
The  contractor  shall  record  all  cases,  procedures,  and 
results  in  the  SDFs. 

(2)  CSCI  Level:  The  contractor  shall  perform 
regression  testing  at  the  CSCI  level.  All  CSCI  level 
functional  and  performance  requirements  that  were 
affected  by  the  change/addition  shall  be  regression 
tested.  This  level  of  testing  shall  be  formally 
demonstrated  to  the  Government  as  a  prerequisite  for 
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successful  conpletion  of  FQT.  Results  shall  be 
docunented  in  the  applicable  test  report. 

(3)  Higher  Level:  If  the  error  occurs  at  a 
level  higher  than  the  CSCI  level,  then  the  contractor 
shall  regression  test  the  failure  up  to  the  level  at 
which  the  “failure  occurred. 

r.  Errors  During  Post  FQT  Testing:  The  contractor 
shall  address  all  software  errors  occurring  subsequent 
to  FQT.  Software  corrective  actions  rectifying  these 
shall  be  regression  tested  to  level  A  and  level  B. 

s.  Software  Test  Reporting:  The  contractor  shall 
document  the  results  of  each  level  of  FQT  lAW  DI-MCCR- 
80017A  and  deliver  the  software  test  report  lAW  the 
associated  CDRL. 

t.  Software  Problem/ Change  Report:  The  contractor 
shall  document  all  anomalies  affecting  software  and/or 
documentation,  regardless  of  source,  lAW  software 
problem/change  reports  (SPCRs)  and  deliver  them  to  the 
Government  lAW  the  associated  CDRL.  The  contractor 
shall  provide  failure  analysis  and  corrective  action 
status  information  for  all  SPCRs  applicable  to  the 
software.  The  contractor  shall  define  a  severity 
classification  for  each  SPCR,  strictly  adhering  to  the 
classification  in  D0D-STD-2167A,  Appendix  C.  Following 
software  baselining  at  the  start  of  FQT,  the  Government 
will  assume  final  approval  authority  over  the  severity 
priority  of  each  SPCR.  The  contractor  shall  maintain  a 
current  computerized  SPCR  database  to  fully  record  and 
aid  in  the  management  and  tracking  of  SPCRs.  The 
contractor  shall  submit  all  updates  of  the  SPCR 
database  in  ASCII  format  via  electronic  media  to  the 
Government  addressees  lAW  the  associated  CDRL. 

u.  Programming  Language:  All  newly  developed 
software  shall  be  generated  in  Ada,  MIL-STD-1815A.  In 
accordance  with  DODD  3405.2  and  DODD  3405.1, 
modifications  to  an  existing  CSCI  which  exceeds  one- 
third  of  the  existing  code  shall  be  considered  newly 
developed  software.  Exceptions  to  this  requirement 
will  require  a  waiver  approved  by  the  Government.  A 
DOD  validated  Ada  compiler (s)  shall  be  used.  Use  of 
assembly  language  must  be  fully  justified  and  requires 
Government  approval.  If  less  than  one-third  of  the 
lines  of  code  are  newly  generated,  then  the  language 
must  meet  DODD  3405.1  requirements. 

V.  Memory  and  Throughput  Growth:  Each  processor  in 

each  delivered  _  (insert  appropriate 

information)  unit  shall  contain  a  50%  growth  capability 
for  both  memory  and  throughput  at  the  time  of  program 
acceptance  by  the  procuring  agency.  Any  requests  for 
deviations  or  waivers  from  this  requirement  will 
require  Government  approval.  The  percentage  of  memory 
or  processor  capability  available  for  growth  will  be 
determined  by  the  following  formula: 
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[1  -  .( used /avail able )  ]  x  100 

w.  Training:  The  contractor  shall  prepare  and 
present  an  informal  training  course  to  LCSEC  personnel 

on  the  use  and  operation  of  _  (insert 

appropriate  information)  system  and  software.  The 
contractor  shall  describe  the  details  of  the  course  in 
the  Computer  Resources  Integrated  Support  Document 
(CRISD)  to  include  but  not  be  limited  to:  technical 
data  for  student  use  and  retention,  lesson  plans  for 
conduct  of  the  course,  and  audio-visual  aids.  The 
training  shall  provide  as  a  minimum,  general  usage 
instructions,  system  initiation,  operation,  monitoring, 
and  security  aspects  (if  applicable) . 

X.  Software  Rights:  The  Government  shall  have 
unlimited  rights  to  all  operating  software  resident  in 

the  _  (insert  appropriate  information).  The 

contractor  shall  identify  any  commercially  available 
support  software  that  has  restricted  rights  in  the 
CRISD.  The  contractor  will  not  include  any  software  in 
the  system  design  and  the  Software  Engineering 
Environment  (SEE)  which  is  proprietary. 
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CHECKLIST: 

SOFTWARE  TIE  STATEMENT  OF  WORK 

Indicate  the  level  at  which  the  Software  T&E  Issue  has 
been  addressed  in  the  subject  SOW.  Assign  a  score 
between  0  ~(not  addressed  in  the  SOW)  and  10  (very  well 
addressed  in  the  SOW) . 


Seftwar*  Z^avel  of 

T&E  Zssu*  AchieTeaeat 

1.  Flow-dovm  of  software  T&E  requirements  to 
third-party  development  activities  and 
subcontractors . 

2.  Data  rights  and  access  to  software  T&E  data. 

3.  Relationship  of  software  T&E  to  the  software 
quality  proces's. 

4.  Requirements  for  demonstration/  evaluation  of 
software  maintainability. 

5.  Interface  of  software  T&E  to  configuration 
management  and  configuration  control 
activities. 

6.  Documentation  and  utilization  of  software 
trouble  reports  (STRs)  which  are  identified 
during  software  T&E. 

7.  Delivery  of  and  utilization  of  software 
development  documentation  by  software  T&E 
activities. 

e.  Involvement  of  software  T&E  in  the  review  and 
audit  process. 

9.  Software  T&E  issues  in  selection  of  the 
development  agent  (SEE  metric  rating). 

10.  Requirements  for  software  test  hooks  of  data 
ports  for  software  T&E  instrumentation. 

11.  Coordinating  IV&V  activities  with  software 
T&E. 

12..  Reporting  of  correlation  between  software  test 
methods  and  objectives  of  the  software  T&E 
program. 

13.  Requirements  for  system  and  software  specifi¬ 
cations  to  identify  testable  requirements. 

14.  Scheduling  of  levels  of  test  to  support 
evaluation  of  software  maturity. 


Figure  B-1.  SOW  Checklist 
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Software 

TftE  Issue 

Level  of 
Achievement 

15. 

Independent  audits  arc  conducted  for  each  step 
of  the  software  development  process. 

16. 

Standards  are  established  for  T&E  of  existing 
designs  and  code  for  re-use  in  new 
applications. 

17. 

Standards  are  applied  to  the  preparation  of 
unit  test  cases. 

18. 

Coding  standards  ars  used. 

19. 

Statistics  on  software  design  errors  are 
available  to  support  T&E. 

20. 

Metrics  are  used  and  applied  logically  and 
usefully. 

21. 

Statistics  on  software  code  and  test  error  are 
plotted  over  time  to  show  trends. 

22. 

Capacity  requirements  (CRU)  are  monitored  for 
computer  memory  utilization. 

23. 

Capacity  requirements  are  monitored  for  CPU 
throughput  and  utilization. 

24. 

Capacity  requirements  are  monitored  for  I/O 
channel  utilization. 

25. 

Security  levels  and  security  accreditation  T&E 
addressed  in  the  SOH/RFP. 

26. 

Test  coverage  is  measured  and  recorded  for 
each  phase  of  software  testing. 

27. 

Action  items  resulting  from  reviews  are 
coordinated  with  T&E  program  and  data  results. 

26. 

Software  Problem/Trouble  Reports  are 
prioritized,  recorded  and  delivered  to  the 
Government . 

29. 

A  formal  mechanism  is  used  for  controlling 
changes  to  the  code. 

30. 

Identification  of  models /simulations  used  for 
T&E  purposes. 

31. 

A  formal  mechanism  is  used  for  configuration 
management  of  the  software  tools  used  in  T6E. 

32. 

Test  beds  and/or  instrumentation  needs  are 
identified  or  required  as  part  of  the  T&E 
program.  Validation/certification  of  these 
assets  is  rec[uired  to  use  them  for  T&E 
purposes . 

0  *  Not  addressed  10  -  Very  well  addressed 


Figure  B-1.  SOW  Checklist  (Cont'd.) 
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Appandiz  C 

Matries  Data  Collaetien 

C-l.  Data  Itam  Dascriptions  (DIDs) 

This  appendix  contains  draft  DD  Forms  1664,  Data  Item 
Descriptions,  to  facilitate  collection  of  the  metrics 
data  described  in  this  part  of  DA  Pamphlet  73-1.  The 
needs  and  resources  of  each  program  will  determine  the 
precise  data  submission  start  times  and  reporting 
frequency  for  each  metric.  See  Chapter  17,  Software 
T&E  Metrics,  for  ^idance  in  this  area.  Start  times, 
frequency  of  submission,  and  other  tailoring 
information  are  included  in  SOWs  via  DD  Forms  1423, 
Contract  Data  Requirements  List.  It  is  recommended 
that  automated  means  of  data  submission  (machine 
readable)  be  employed  for  the  majority  of  the  data 
because  of  its  potentially  large  volume,  in  some  cases. 
It  may  also  be  desirable  to  combine  the  submission  of 
several  metrics  reports  if  they  occur  with  the  same 
reporting  frequency.  Note:  cost  metric  data  can  be 
obtained  from  cost  reports  submitted  by  means  of  DI-F- 
6000C  or  DI-F-6010A  if  the  SOW  or  DD  Form  1423  states 
that  software  elements  as  described  in  this  part  of  DA 
Pamphlet  73-1  are  to  be  included  in  the  report.  A 
sample  cost  metric  DID  for  software  elements  alone  is 
provided  in  this  appendix  for  those  tasks  which  may  not 
be  required  to  report  via  the  two  DIDs  above. 
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PROPOSED  DRAFT.  DO  NOT  OSE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


OUB  No.  0704^188 


Z  TriLE 


SOFTWARE  COST  METRIC  DATA  REPORT 


3.  DESCRiPTiON/PURPOSE 


1.  IDENTIFCATON  NUMBER 
DI-XXXX-XXXXX 


thi  -noction.  ov.r 


I  ^  primary  RESPONSaiLfTY  (OPR) 


6a  OTC  APPUCABU  66.  GIOEP  APPLCABLE 


7.  APPLlCATIOri'INTERRELATlONSHlP - - - - - - — ^ - * - - - — 

contains  tha  format  and  contant  praparation  instructions  for 
softwara  cost  matric  data  rasulting  from  tha  work  task  dascribad  by  AR  73-1. 

7.2  This  DID  may  ba  appliad  to  any  softwara  davalopmant  or  maintananca  contract. 


8.  APPROVALLWITATION 


9a  APPLICABIE  FORMS 


9b.  AMSC  NUMBER 


10.  PREPARATON  INSTRUCTIONS  - - - -  ‘  - - 

Docur^antP.  Tha  applicabla  issua  of  tha  documants  citad  harain, 

and^rfvrLftnr^h!!??^K''*^  cUtas  and  datas  of  any  applicabla  amandmants,  noticas, 
ana  ravisions  shall  ba  as  spacifiad  in  tha  contract. 

rjrrrj:  Reguirementg.  Th«  Software  Cost  Matric  Data  Raport  shall  ba 
ccffiprisao  of  introductory  summary  matarial  and  itamirad  matric  data.  Summarv 

Item^rad  5  P«P«r  or  alactronic  madia. 

Item-.2eQ  data  shall  ba  suppliad  on  alactronic  madia  or  8  1/2  by  11  inch  paoar 

such  that  It  may  ^  raadily  compatible  with  an  alactronic  dataLsa  structufa.  Tha 

*ach  typa  of  data  in  tha  raport  is  dascribad  in  its  corraspondino 
Contant  Raquiramant  subparagraph  balow.  ^  oing 

7ailgrin,g  IngtxuctlPns.  in  tha  avant  that  a  paragraph  or  subparagraph  has 
out,  a  statamant  to  that  affact  shall  ba  added  directly  following 
tha  heading  of  each  such  (sub) paragraph.  ^  ’ 

10.2.2  EardcpPY  rgrmat.  This  document  may  ba  printed  on  one  side  or  both  sides 
of  each  page  (singla-sidad/doubla-sidad) .  All  printed  pages  shall  contain  tha 
document  control  number  and  publication  data  cantered  at  the  top  of  tha  page. 

Paragraphs  and  Subpragraphs.  Any  paragraph  or  subparagraph  in 
this  Dp  starting  with  tha  phrase  "this  (sub) paragraph  shall..."  ma?  ^written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability.  ^ 


11.  DiSTRlBl/TION  statement 


(Continued  on  Page  2) 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  reouirementg.  The  Software  Cost  Metric  Data  Report  shall 
contain  the  following  information: 

10.3.1  Title  page.  This  page  shall  contain  the  name  apd  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Cost  metric  record.  For  each  activity  type  the  following  information 
shall  be  supplied.  The  format  for  cost  metric  data  is  shotm  in  Table  I. 

a.  DATA^DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  ””That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYYY/MM/DD. 

b.  SYSTEM_NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI  NAME  -  The  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  3ata  applies.  No  value  is  required  for  this  field  when  reporting 
system/project  level  costs. 

d.  ACTIVITY_TYPE  -  The  type  of  effort  or  product  associated  with  the 
collected  dataT  Acceptable  values  for  CSCI  level  reporting  are:  requirements 
analysis,  design,  code  and  unit  testing.  Computer  Software  Component  (CSC) 
integration  and  testing,  formal  qualification  testing,  and  software  problem 
change  report  resolution.  Acceptable  values  for  system/project  level 
reporting  are:  CSCI  integration  and  testing,  software  engineering  management, 
software  quality  assurance,  software  configuration  management,  verification 
and  validation,  tools,  new  equipment  and  facilities,  software  data  and  project 
totals.  Definitions  for  the  work  performed  in  each  activity  type  are  provided 
in  MIL-HDBK-WBS.SW,  Second  Draft,  dated  1  October  1991. 

e.  BCWS  -  The  total  number  of  dollars  that  had  been  budgeted  for  the  work 
scheduled  to  be  accomplished  for  the  CSCI  as  of  reporting  period.  Acceptable 
values  are  in  the  range  0  through  999999999.99. 

f.  BCWP  -  The  total  number  of  dollars  budgeted  for  the  work  actually 
performed  on  the  CSCI  as  of  reporting  period.  Acceptable  values  are  in  the 
range  0  through  999999999.99. 

g.  ACWP  •  The  total  number  of  dollars  which  was  actually  spent  for  the  work 
done  on  the  CSCI  as  of  this  reporting  period.  Acceptable  values  are  in  the 
range  0  through  999999999.99. 
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TABLE  I .  ■ Cost  Metric  Date  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2/ 

DATA_DATE 

Character 

1/ 

10 

SYSTEM^NAME 

Character 

20 

CSCI_NAME 

Character 

15 

ACTIVITY^TYPE 

Character 

15 

BCWS 

Numeric 

1/ 

12 

2 

BCWP 

Numeric 

12 

2 

ACWP 

Numeric 

12 

2 

Number  of  numerals  after  the  decimal  point. 

Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

Numerals,  decimal  point  or  negative  sign  (-). 


1/  Decimal  - 
2./  Character  - 

2/  Numeric  - 


) 
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PROPOSED  DRAFT.  DO  MOT  OSE  FOR  ACQOISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


Fom  Afiprxfit9d 
0MB  No.  0704^138 


SOFTWARE  SCHEDULE  METRIC  DATA  REPORT 


1.  OENTiFCATiON  NUMBER 
DI-XXXX-XXXXX 


3.  OESCRIPTION'PURPOSE 

3.1  This  metric  data  is  used  to  track  key  events  or  deliveries  in  the  software 
development  program. 


4.  APPROVAL  DATE  1 1  OFFCE  OF  PRIMARY  RESPONSIBILITY  (OPR) 

{YYMMDOi  I 


te.  OTIC  APPLCABLE  6D.  610EP  APPLICABLE 


7.  APPLiCATlONirVTERRELATlONSHiP 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  schedule  metric  data  resulting  from  the  work  task  described  by  AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


e.  approval  LIMITATION 


Sa  APPLCABLE  FORMS 


Ob.  AMSC  NUMBER 


1C.  PREPARATION  INSTRUCTIONS 

10.1  Reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  FcrrrAt  Requirements.  The  Software  Schedule  Metric  Data  Report  shall  be 
comprised  of  introductory  summary  material  and  itemized  metric  data.  Summary 
material  shall  be  prepared  on  6  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
that  it  ray  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

10.2.1  Tailoring  Inst ructions*  In* the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  Hardcopy  Formats  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) •  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  Subpragragbs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  **this  (sub) paragraph  shall.  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 
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10.3  Content  requirements.  The  Software  Schedule  Metric  Data  Report  shall 
contain  the  following  information: 

10.3.1  Title  page.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  sygtem,  name  of  sponsoring  or  issuing  organisation,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period (s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Schedule  metric  record.  For  each  milestone  for  which  schedule  data  is 
required,  the  following  information  shall  be  supplied.  The  format  for 
schedule  metric  data  is  shown  in  Table  I. 

a.  0ATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  "That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  TYTY/MH/DD. 

b.  SYSTEM_NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  EVENT_NAME  -  The  name  of  the  milestone,  deliverable,  event  or  activity 
this  record'describes.  Examples  of  events  are  formal  system  reviews,  testing 
events,  and  software  data  product  deliveries. 

d.  PLAN_START_DATE  •  The  date  on  which  the  event  is  planned  to  start.  The 
date  shall  be  expressed  as  YYYY/MM/DD. 

e.  PLAN  END_DATE  **  The  date  on  which  the  event  is  planned  to  be  completed. 
The  date  sHall~be  expressed  as  YYYY/MM/DD. 

f.  ACTUAL_START_DATE  -  The  date  the  event  actually  began.  The  date  shall 
be  expressed~as  YYYY/MM/DD. 

g.  ACTUAL_END_DATE  -  The  date  the  event  actually  completed.  The  date  shall 
be  expre88ed~as  YYYY/MM/DD. 


TABLE  I.  Schedule  Metric  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  1/ 

DATA_DATE 

Character  2/ 

10 

SYSTEM^NAME 

Character 

20 

EVENT_NAME 

Character 

12 

PLAN_START_DATE 

Character 

10 

PLAN_END_DATE 

Character 

10 

ACTUAL_START_DATE 

Character 

10 

ACTUAL_END_DATE 

Character 

10 

X!  Decimal  --  Number  of  numerals  after  the  decimal  point. 

2.1  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 


Page  2  of  2  Pages 


C-« 


30  8«pt«Bber  1992 


DA  Pan  73-1,  Part  savan  (Draft) 


PROPOSED  DRAFT.  DO  NOT  USE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


fom  Afipto¥0d 
OUB  No.  0704’01$3 


Z  TfTlE 


COMPUTER  RESOURCE  UTILIZATION  METRIC  DATA  REPORT 


1.  lOENTlFlCATON  NUMBER 


DI-XXXX-XXXXX 


3.  DESCRIPTION/PURPOSE 


3.1  This  metric  data  is  used  to  track  projected  and  measured  utiliration  of 
Central  Processing  Units  (CPUs),  Input/Output  channels,  RAM  memory  and  mass 
storage  devices. 


4.  APPROVAL  DATE 
{rYMMDO} 


5.  Off  CE  Of  PRIMARY  RESPONSSIUTY  (OPR) 


6e  DTC  APPLICABLE  |  6D.  GiOEP  APPLCABLE 


7.  APPLICATION-INTERRELATIONSHIP 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  computer  resource  utilization  metric  data  resulting  from  the  work  task 
described  by  AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract, 


B.  approval  LIMITATION 


te.  applicable  FORMS 


St>.  AMSC  NUMBER 


10.  preparaton  instructions 


ReftrciiCe  Docunients*  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

FgrrAt  Resr^irementg.  The  Computer  Resource  Utilization  Metric  Data  Report 
shall  be  comprised  of  introductory  summary  material  and  itemized  metric  data. 
Summary  material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic 
media.  Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch 
paper  such  that  it  may  be  readily  compatible  with  an  electronic  database 
structure.  The  structure  for  each  type  of  data  in  the  report  is  described  in  its 
corresponding  Content  Requirement  subparagraph  below. 

^0.2.1  Tail or In Q  Instructions.  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2'  Hardcopy  Format#  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  Subora graphs.  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  •‘this  (sub) paragraph  shall... •*  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Confd.) 

10.3  Content  requirements.  The  Computer  Resource  Utilisation  Metric  Data 
Report  shall  contain  the  following  information: 

10.3.1  Title  page.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraphs  10.3.3  and  10.3.4.  Significant  or  unusual  circumstances 
affecting  the  preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Computer  resource  utilization  (CRm  metric.  For  each  target  computer 
resource  in  the  system  information  shall  be  supplied  in  accordance  with  the 
paragraphs  below. 

10.3.3.1  Central  Processing  Unit  fCPU^  utilization  record.  For  each  CPU  in 
the  system  the  following  data  shall  be  supplied.  The  format  for  CPU 
utilization  data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  ~That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  yyyy/MM/DD. 

b.  sySTEM_NAM£  -  N2Lme  of  the  system  to  which  this  data  applies. 

c.  CPU_1D  -  A  unique  identifier  for  the  CPU. 

d.  PCNT_CPU_TGT  -  The  target  upper  bound  utilization  value  for  the  CPU. 

This  is  the  desired  maximum  value  for  this  CPU's  utilization  expressed  as  a 
percentage  of  total  capacity  of  the  CPU.  Acceptable  values  are  in  the  range  0 
through  100. 

e.  PCNT_CPU_PROJ  -  The  projected  capacity  utilization  value  for  the  CPU. 
This  is  the  estimated  percentage  of  maximum  utilization  expected  at  delivery. 
Acceptable  values  are  in  the  range  0  through  100. 

f.  PCNT  CPU_ACT  -  The  measured  value  of  CPU  capacity  utilized  during  peak 
operationaT  loading  periods  expressed  as  a  percentage  of  total  capacity  of  the 
CPU.  Acceptable  values  are  in  the  range  0  through  100. 

g.  COMP_TyPE  -  Identification  as  to  whether  measurements  were  taken  on  the 
actual  target  machine  or  a  software  test  environment  configuration. 

Acceptable  values  are  T  (target)  or  H  (host). 

10.3.3.2  Input /Output  (I/O)  channel  utilization  record.  For  each  I/O  channel 
in  the  system  the  following  data  shall  be  supplied.  The  format  for  I/O 
channel  utilization  data  is  shown  in  Table  II. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  ~That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  yyyy /MM/00. 

b.  SySTEM_NAHE  -  Name  of  the  system  to  which  this  data  applies. 

c.  CHAN_ID  -  A  uniques  identifier  for  the  I/O  channel. 

d.  PCNT_CHAN  TGT  -  The  target  upper  bound  utilization  value  for  the  I/O 
channe] .  This  Is  the  desired  maximum  value  for  this  channel's  utilization 
expressed  as  a  percentage  of  total  capacity  of  the  channel.  Acceptable  values 
are  in  the  range  0  through  100. 
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e.  PCNT_CHAN^PROJ  -  The  projected  capacity  utilization  value  for  the  I/O 
channel.  This  Te  the  eatimated  percentage  of  maxinun  utilization  expected  at 
delivery.  Acceptable  values  are  in  the  range  0  through  100. 

f.  PCNT_CHAN_ACT  -  The  measured  value  of  I/O  channel  capacity  utilized 
during  peak  operational  loading  periods  expressed  as  a  percentage  of  total 
capacity  of  the  channel.  Acceptable  values  are  in  the  range  0  through  100. 

g.  COMP_TyPE  -  Identification  as  to  whether  measurements  were  taken  on  the 
actual  target  machine  or  a  software  test  environment  configuration. 

Acceptable  values  are  T  (target)  or  H  (host). 

10.3.3.3  Random  Access  Memory  (RAM^  utilization  record.  Por  each  RAM  in  the 
system  the  following  data  shall  be  supplied.  The  format  for  RAM  utilization 
data  is  shown  in  Table  III. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  "That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YTTY/MM/OD. 

b.  SYSTEM_NAKE  -  Name  of  the  system  to  which  this  data  applies. 

c.  RAM_ID  -  A  unique  identifier  for  the  RAM  device. 

d.  RAM_CAPACITY  •  The  capacity  of  the  RAM  in  bytes.  Acceptable  values  are 
in  the  range  0  through  99999999999. 

e.  RAM_TGT  -  The  target  upper  bound  memory  usage  value,  in  bytes,  for  the 
RAM  device.  This  is  the  desired  maximum  value  for  this  RAM  device's  usage. 
Acceptable  values  are  in  the  range  0  through  99999999999. 

f.  RAM_PROJ  -  The  projected  capacity  utilisation  value  for  the  RAM  device. 
This  is  the  estimated  maximum  number  of  bytes  to  be  utilized  at  delivery. 
Acceptable  values  are  in  the  range  0  through  99999999999. 

g.  RAM_ACT  -  The  measured  value  of  RAM  utilized  during  peak  operational 
loading  periods,  in  bytes.  Acceptable  values  are  in  the  range  0  through 
99999999999. 

h.  COMP_TYPE  -  Identification  as  to  whether  measurements  were  taken  on  the 
actual  target  machine  or  a  software  test  environment  configuration. 

Acceptable  values  are  T  (target)  or  H  (host). 

10.3.3.4.  Mass  storage  utilization  record.  For  each  mass  storage  device  in 
the  system  the  following  data  shall  be  supplied.  The  format  for  mass  storage 
utilization  data  is  shown  in  Table  IV. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  ~That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYYY /MM/00. 

b.  sySTEM_NAME  -  Name  of  the  system  to  which  this  data  applies, 

e.  STOR_IO  -  A  unique  identifier  for  the  mass  storage  device. 

d.  STOR_CAPACITy  -  The  capacity  of  the  storage  in  bytes.  Acceptable 
values  are~in  the  range  0  through  99999999999. 

e.  STOR_TGT  -  The  target  upper  bound  mass  storage  usage  value,  in  bytes, 
for  the  storage  device.  This  is  the  desired  maximum  value  for  this  device's 
usage.  Acceptable  values  are  in  the  range  0  through  99999999999. 

f.  STOR_PROJ  -  The  projected  capacity  utilization  value  for  the  mass 
storage  device.  This  is  the  estimated  maximum  number  of  bytes  to  be  utilized 
at  delivery.  Acceptable  values  are  in  the  range  0  through  99999999999. 
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g.  STOR  ACT  -  Th«  maasurad  valua  of  tha  storaga  davica  utilizad  during  peak 

operational  loading  periods,  in  bytes.  Acceptable  values  are  in  the  range  0  ) 

through  99999999999. 

h.  COMP_TyPE  -  Identification  as  to  whether  measurements  were  taken  on  the 
actual  target  machine  or  a  software  test  environment  configuration. 

Acceptable  values^are  T  (target)  or  H  (host). 

10.3.4  Software  memory  allocation  record.  For  each  Computer  Software 
Configuration  Item  (CSCI)  in  the  system  the  following  data  shall  be  supplied. 

The  format  for  software  memory  allocation  data  is  shown  in  Table  V. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  TYTY/MM/DD. 

b.  SYSTEM^NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI__NAME  -  The  Computer  Software  Configuration  Item  (CSCI)  name  to 

which  the  data  applies.  .  . 

d.  CSC1_RW1  ALLOC  -  The  RAM  allocation  in  bytes  budgeted  for  this  CSCI  in 
accordance  witE  its  Software  Requirements  Specification.  Acceptable  values 
are  in  the  range  0  through  99999999999. 

e.  CSCI_RAM_USAGE  -  The  actual  amount  of  RAM  used  by  the  CSCI,  in  bytes. 

Acceptable  values  are  in  the  range  0  through  99999999999. 

f.  CSCI_STOR_ALLOC  -  The  mass  storage  allocation  in  bytes  budgeted  for  this 
CSCI  in  accordance  with  its  Software  Requirements  Specification.  Acceptable 
values  are  in  the  range  0  through  99999999999. 

g.  CSCI_STOR_USAGE  -  The  actual  amount  of  mass  storage  used  by  the  CSCI,  in 
bytes.  AcceptaEle  values  are  in  the  range  0  through  99999999999. 

h.  COMP_TYPE  -  Identification  as  to  whether  measurements  were  taken  on  the 
actual  target  machine  or  a  software  test  environment  configuration. 

Acceptable  values  are  T  (target)  or  H  (host). 


TABLE  I.  Central  Processing  Unit  Utilization  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2./ 

DATA_DATE 

Character 

i/ 

10 

SYSTEM^NAME 

Character 

20 

CPU_ID 

Character 

12 

PCNT_CPU_TCT 

Numeric 

1/ 

3 

0 

PCNT_CPU_PROJ 

Numeric 

3 

0 

PCNT_CPU_ACT 

Numeric 

3 

0 

C0MP_TYPE 

Character 

1 

i/  Decimal  - 
2/  Character  - 

3/  Numeric  - 


Number  of  numerals  after  the  decimal  point. 

Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

Numerals,  decimal  point  or  negative  sign  (-). 
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TABLE  II.  Input/Output  Channel  Utilization  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal 

DATA^DATE 

Character 

10 

SYSTEM^NAME 

Character 

20 

CHAN^ID 

Character 

12 

PCNT_CHAN_TGT 

Numeric 

3 

0 

PCNT_CHAN_PROJ 

Numeric 

3 

0 

PCNT_CHAN_ACT 

Numeric 

3 

0 

COKP^TYPE 

Character 

1 

TABLE  III.  Random  Access  Memory  Utilization  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal 

DATA^DATE 

Character 

10 

SYSTEM_NAME 

Character 

20 

RAM_ID 

Character 

12 

RAM_CAPACITY 

Numeric 

11 

0 

RAM_TGT 

Numeric 

0 

RAM^PROJ 

Numeric 

0 

RAM_ACT 

Numeric 

0 

COMP^TYPE 

Character 

TABLE  IV.  Mass  Storage  Utilization  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal 

DATA^DATE 

Character 

10 

SYSTEMJ^AME 

Character 

20 

STOR_ID 

Character 

12 

STOR_CAPACITY 

Numeric 

11 

0 

STOR^TGT 

Numeric 

0 

STOR_PROJ 

Numeric 

B9 

0 

STOR^ACT 

Numeric 

0 

COMP  TYPE 

Character 

C-11 
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TABLE  V.  Software  Memory  Allocation  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal 

DATA^DATE 

Character 

10 

SYSTEM^NAME 

Character 

20 

CSCI^NAME 

Character 

15 

CSCI_RAM_ALLOC 

Numeric 

11 

0 

CSCI_RAM_USAGE 

Numeric 

0 

CSCI_STOR_ALLOC 

Numeric 

mam 

0 

CSCI_STOR_USAGE 

Numeric 

0 

COMP^TYPE 

Character 

■■1 
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PROPOSED  DRAFT.  DO  NOT  DSE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


OidBNo.  0704^1B8 


SOFTWARZ  ENGINEERING  ENVIRONMENT  METRIC  DATA  REPORT 


1.  CeMTlFICATON  NUMBER 
DI-XXXX-XXXXX 


3.  DESCRIPTIONPURPOSE 

3.1  This  metric  data  provides  a  measure  of  the  capability  of  a  contractor  to  use 
modern  engineering  techniques  in  the  software  development  process. 


4.  APPROVAL  DATE  15.  OF FCE  OF  PRIMARY  RESPONSIBILITY  (OPR) 

{YYIAMDDi 


6a.  OTC  APPLICABLE  I  6t>.  GIOEP  APPLICABLE 


7.  APPLiCATlON'iNTERREtATlONSHiP 

7.1  This  DID  contains  the  format  and  content  preparation  instructions  for  the 
software  engineering  environment  metric  data  resulting  from  the  work  task 
described  by  AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8.  APPROVAL  LIMITATION 


9a  applicable  FORMS 


99.  AMSC  NUMBER 


10.  PREPARATION  INSTRUCTIONS 

10.1  Reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  Requirements .  The  Software  Engineering  Environment  Metric  Data 
Report  shall  be  comprised  of  introductory  summary  material  and  itamized  metric 
data.  SunvTiary  material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic 
media.  Itemized  data  shall  be  supplied  on  electronic  media  or  6  1/2  by  11  inch 
paper  such  that  it  may  be  readily  compatible  with  an  electronic  database 
structure.  The  structure  for  each  type  of  data  in  the  report  is  described  in  its 
corresponding  Content  Requirement  subparagraph  below. 

10.2.1  Tailoring  Instructions.  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  ^ardcopy  Format.  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-aided) .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  data  centarad  at  tha  top  of  tha  page. 

10.2.3  Multiple  Paragraphs  and  Suboraoraphs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  "this  (sub) paragraph  shall..."  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 


(Continued  on  Page  2) 


11.  DiSTRlBLfTlON  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  reouiremente.  The  Software  Engineering  Environment  Metric  Data 
Report  shall  contain  the  following  information: 

10.3.1  Title  pane.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period (s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Software  Enoineerino  Environment  fSEE\  metric  record.  For  each 
contractor  on  which  a  software  maturity  level  assessment  is  performed  the 
following  information  shall  be  supplied.  The  format  for  software  engineering 
environment  metric  data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  TTYT/MM/DD. 

b.  CONTRCTR_NM  -  The  name  of  the  contractor  that  was  evaluated. 

c.  SYSTEM_NAH£  -  Name  of  the  system  to  which  this  data  applies. 

d.  MAT_LEVEL  -  The  process  maturity  level  of  the  contractor.  Acceptable 
values  are  in  the  range  1  through  5. 

e.  MAT_DATE  -  The  date  the  process  maturity  level  was  assigned  to  the 
contractor.  The  date  shall  be  expressed  as  YYyy/MM/DD. 


TABLE  I.  Software  Engineering  Environment  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  1/ 

DATA^DATE 

Character 

2/ 

10 

CONTRCTR^NM 

Character 

15 

SYSTEM^NAME 

Character 

20 

MAT^LEVEL 

Numeric 

1/ 

1 

0 

MAT^DATE 

Character 

10 

i,/  Decimal  -  Number  of  numerals  after  the  decimal  point. 

1/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

1/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (•>). 
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PROPOSED  DRAFT.  DO  NOT  USE  FOR  ACOOISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


FormAppwd 
OM0M9. 0704^^aa 


2.  TITLE 


SOFTWARE  REQUIREMENTS  TRACEABILITY  METRIC  DATA  REPORT 


1.  IDENTTIFCATOM  NUMBER 
DI-XXXX-XXXXX 


3.  DESCRIPT lON'PURPOSE 

3,1  This  metric  data  is  usad  to  track  ths  adhsranca  of  softwara  products  to 
their  requirements  at  various  davalopmantal  levels. 


4.  approvaldate 

{YYMMDDi 


5.  OFFCE  OF  PRIMARY  RESPONSIBILITY  (OPR) 


6e.  DTIC  APPtCABLE  6b.  GIDEP  APPLICABLE 


7.  APPLICATION'INTERRELATIONSHIP 

7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  requirements  traceability  metric  data  resulting  from  the  work  task 
described  by  AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8.  APPROVAL  LIMITATION 


applicable  FORMS 


9b.  AMSC  NUMBER 


10.  preparation  INSTRUCTIONS 


Reftrencs  Dgcumgnts*  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

EgxrJi t  Re qu i r emcat S «  The  Software  Requirements  Traceability  Metric  Data 
Report  shall  be  comprised  of  introductory  summary  material  and  itemized  metric 
data.  Summary  material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic 
meoia.  Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch 
paper  such  that  it  may  be  readily  compatible  with  an  electronic  database 
structure.  The  structure  for  each  type  of  data  in  the  report  is  described  in  its 
corresponding  Content  Requirement  subparagraph  below. 

10.2.1  Tailoring  lDgtructiQng>  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  '  Kardcopy  Foiinat..>  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided).  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  Subpxagxapha>  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  -this  (sub) paragraph  shall...-  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 


(Continued  on  Page  2) 


11.  DISTRIBUTIC'  ' STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  reouirementB.  The  Software  Requirements  Traceability  Metric 
Data  Report  shall  contain  the  following  information: 

10.3.1  Title  pace.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10*3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraphs  10.3.3  and  10.3.4.  Significant  or  unusual  circumstances 
affecting  the  preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Software,  retirements  traceability  metric  record.  For  each  Computer 
Software  Configuration  Item  (CSCI)  in  the  system  the  following  information 
shall  be  supplied.  The  format  for  software  requirements  traceability  metric 
data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYYY/MM/DD. 

b.  SYSTEM_NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  -  The  Computer  Software  Configuration  Item  name  to  which  the 
data  applies. 

d.  MILESTONE  -  The  name  of  the  event  or  activity  this  to  which  this  data 
applies. 

e.  VERSION_ID  -  The  configuration  control  version  identification  of  the 
CSCI  that  is  being  reported. 

f.  NUM_SRS_REQ  -  The  number  of  Software  Requirements  Specification  (SRS) 

requirements  for  this  CSCI.  Acceptable  values  are  in  the  range  0  through 
99999.  ^ 


g.  NUM_SRS_REQ_NOT_urD  -  The  number  of  SRS  recpiirements  in  this  CSCI  that 
are  not  traceable  to  the  UFD.  Acceptable  values  are  in  the  range  0  through 
99999. 


h.  NUM_UFD_SW_REQ_SRS  —  The  number  of  UFD  software  requirements  traceable 
to  the  SRS  for  this  CSCI.  Acceptable  values  are  in  the  range  0  through  99999. 

i.  NUM__SSS_SW__REQ_SRS  -  The  number  of  SSS  software  related  requirements 

traceable  to  the  SRS  for  this  CSCI.  Acceptable  values  are  in  the  range  0 
through  99999.  ^ 

j.  NUM_^SRS_REQ  CSCI  -  The  number  of  the  CSCI's  SRS  requirements  that  are 
traceable  to  the  Hocumented  design  of  the  CSCI.  Acceptable  values  are  in  the 
range  0  through  99999. 

k.  NUM_SRS  REQ_CSC  -  The  number  of  the  CSCI’s  SRS  requirements  traceable  to 
the  documenteS  design  of  the  CSCI's  Computer  Software  Components  (CSCs). 
Acceptable  values  are  in  the  range  0  through  99999. 

l.  NUH_SRS  REQ_CSU  -  The  number  of  the  CSCI’s  SRS  requirements  traceable  to 
the  documented  design  of  the  CSCI’s  Computer  Software  Units  (CSUs). 

Acceptable  values  are  in  the  range  0  through  99999. 

ro.  NUM_SRS_REQ_CODE  -  The  number  of  the  CSCI's  SRS  requirements  traceable 
to  the  computer  program  code  comprising  the  CSCI.  Acceptable  values  are  in 
the  range  0  through  99999. 
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n.  NUM_SRS_REQ_TSCS  -  The  number  of  the  CSCI’s  SRS  requirements  covered  by 
one  or  more  documented  test  case.  Acceptable  values  are  in  the  range  0 
through  99999. 

o.  NUM_IRS_REQ_CSCI  -  The  number  of  IRS  requirements  that  are  traceable  to 
the  documented  design  of  the  CSCI.  Acceptable  values  are  in  the  range  0 
through  99999. 

p.  KUM_IRS_REQ_CSC  -  The  number  of  IRS  requirements  traceable  to  the 
documented  de8ign~of  the  CSCI's  Computer  Software  Components  (CSCs). 

Acceptable  values  are  in  the  range  0  through  99999. 

q.  NUM__IRS_REQ_CSU  -  The  number  of  IRS  requirements  traceeible  to  the 
documented  design~of  the  CSCI's  Computer  Software  Units  (CSUs).  Acceptable 
values  are  in  the  range  0  through  99999. 

r.  NUM_1RS_REQ_C0DE  -  The  number  of  IRS  requirements  traceable  to  the 
computer  program  code  comprising  the  CSCI.  Acceptable  values  are  in  the  range 
0  through  99999. 

8.  NUM_IRS_REQ__TSCS  -  The  number  of  IRS  requirements  covered  by  one  or  more 
documented  test  case.  Acceptable  values  are  in  the  range  0  through  99999. 

10.3.4  Overall  requirements  traceability  metric  record.  One  record  of  this 
type  shall  be  submitted  for  each  reporting  period.  The  format  for  overall 
requirements  traceability  metric  data  is  shown  in  Table  II. 

a.  DATA^OATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  'That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  TYTY/MM/DD. 

b.  SYSTEM_NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  MILESTONE  -  The  name  of  the  event  or  activity  this  to  which  this  data 
applies. 

d.  NUM_ORD_REQ  -  The  number  of  Operational  Requirements  Document  (ORD) 
requirements  for  the  system.  Acceptable  values  are  in  the  range  0  through 
99999. 


e.  NUM_UFD  REQ  -  The  number  of  User  Functional  Description  (UFD) 
requirements  ?or  the  system.  Acceptable  values  are  in  the  range  0  through 
99999. 

f.  NUM_UFD  SW_REQ  -  The  number  of  UFD  software  requirements  for  the  system. 
Acceptable  values  are  in  the  range  0  through  99999. 

g.  NUM_SSS_REQ  -  The  number  of  system  level  specification  (e.g. 
System/Segment  Specification,  SSS)  requirements  for  the  system.  Acceptable 
values  are  in  the  range  0  through  99999. 

h.  NUM_SSS__SH_REQ  -  The  number  of  software  related  requirements  in  the  SSS 
for  the  system.  ~Acceptable  values  are  in  the  range  0  through  99999. 

i.  NUM_IRS_REQ  -  The  number  of  software  interface  requirements  for  the 
system.  Acceptable  values  are  in  the  range  0  through  99999. 

j.  NUM_ORD_REQ_UFD  -  The  number  of  ORD  requirements  which  are  traceable  to 
the  UFD.  ~Acceptable  values  are  in  the  range  0  through  99999. 

k.  NUM_UFD_REQ_ORD  -  The  number  of  UFD  requirements  traceable  to  the  ORD. 
Acceptable  vaTues~ire  in  the  range  0  through  99999. 

l.  NUM_UFD  REQ_SSS  -  The  number  of  UFD  requirements  traceable  to  the  SSS. 
Acceptable  vaTues  are  in  the  range  0  through  99999. 
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in.  NIJM_UFD_SW_REQ  IRS  -  The  number  of  UFD  software  requirements  traceable 
to  an  IRST  AcceptabXe  values  are  in  the  range  0  through  99999. 

n.  NUM_IRS_REQ_NOT__UFD  -  The  number  of  IRS  requirements  that  are  not 
traceable~to  the  UFD.”  Acceptable  values  are  in  the  range  0  through  99999. 

o.  KUM_SSS_SW_REQ_IRS  -  The  number  of  SSS  software  related  requirements 
that  are  traceable  to  an  IRS.  Acceptable  values  are  in  the  range  0  through 
99999. 


TABLE  I.  Software  Requirements  Traceability  Metric  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2/ 

DATA^DATE 

Character  2/ 

10 

SYSTEM^NAME 

Character 

20 

CSCI_NAME 

Character 

15 

MILESTONE 

Character 

12 

VERSION^ID 

Character 

15 

NUM_SRS_REQ 

Numeric  2/ 

5 

0 

NUM_SRS_REQ_NOT_UFD 

Numeric 

5 

0 

NUM_UFD_SW_REQ_SRS 

Numeric 

5 

0 

NUM_SS S_S W_REQ_SRS 

Numeric 

5 

0 

NUM_SRS_REQ_CSCI 

Numeric 

5 

0 

NUM_SRS_REQ_CSC 

Numeric 

5 

0 

NUM_SRS_REQ_CSU 

Numeric 

5 

0 

NUM_SRS_REQ_CODE 

Numeric 

5 

0 

NUM_SRS_REQ_TSCS 

Numeric 

5 

0 

NUM_IRS_REQ_CSCI 

Numeric 

5 

0 

NUM_IRS_REQ_CSC 

Numeric 

5 

0 

NUM_IRS_REQ_CSU 

Numeric 

5 

0 

NUM_IRS_REQ_CODE 

Numeric 

5 

0 

NUM  IRS  REQ  TSCS 

Numeric 

5 

0 

1/  Decimal  >  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

2/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (-). 
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TABLE  II.  Overall  Requirements  Traceability  Metric  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal 

DATA_DATE 

Character 

10 

SYSTEM^NAME 

Character 

20 

MILESTONE 

Character 

12 

NUM_ORD_REQ 

Numeric 

5 

0 

:  NUM_UFD_REQ 

Numeric 

5 

0 

;  NUM_UFD_SW_REQ 

Numeric 

5 

0 

NUM_SSS_REQ 

N\2meric 

5 

0 

mJM_SSS_SW_REQ 

N\uneric 

5 

0 

NUM_IRS_REQ 

Numeric 

5 

0 

NUM_ORD_REQ_UFD 

Numeric 

5 

0 

'  NUM_UFD_REQ_CRD 

Numeric 

5 

0 

NUM_UFD_REQ_SSS 

Numeric 

5 

0 

NUM_UFD_SW_REQ_IRS 

Numeric 

5 

0 

;  NUM_IRS_REQ_NOT_UFD 

' Numeric 

5 

0 

NUM_SSS_SW_REQ_IRS 

Numeric 

5 

0 
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PROPOSED  DRAFT.  DO  NOT  OSE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


fpem  Appra¥0d 
OklB  No.  0704^138 


2.  Tnu 

SOFTWARE  REQUIREMENTS  STABILITY  METRIC  DATA  REPORT 


[I.  CENTJFiCATtON  NUMBER 
DI-XXXX-XXXXX 


3.  DESCRIPTION  PURPOSE 


3.1  This  metric  data  is  used  to  indicate  the  degree  to  which  changes  in  the 
software  requirements  affect  the  development  effort. 


4.  approvaldate 
(VrMMDOi 


5-  OFFICE  OF  PRIMARY  RESPONSSUfTY  (OPR) 


6a.  OTC  APPLICABLE 


6b.  GIOEP  APPLICABLE 


7.  APPUCATIONINTERRELATIONSHIP 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  requirements  stability  metric  data  resulting  from  the  work  task 
described  by  AR  73-1. 


7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8. 


APPROVAL  LIMITATION 


te.  applicable  FORMS 


9b. 


10.  PREPARATION  INSTRUCTIONS 


AMSC  NUMBER 


Reference  Dgcuments.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

^^•2  r.crnaC  Reguirements*  The  Software  Requirements  Stability  Metric  Data  Report 
shall  be  comprised  of  introductory  summary  material  and  itemized  metric  data. 
Summary  material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic  media. 
Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
t.hat  it  may  be  readily  conT>atible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

^0.2.1  IflilQrinq...Iiigtruction8 .  in  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2,2^  ii&xdcgpy  Jozmat#  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple ...Bajaoraphs^And  Suborapraohs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  "this  (sub) paragraph  shall..."  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 


(Continued  on  Page  2) 

11.  DISTRIBUTION  STATEMENT  - 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PRBPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  reouirements.  The  Software  Requirements  Stability  Metric  Data 
Report  shall  contain  the  following  information: 

10.3.1  Title  page.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period (s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Requirements  stability  metric  record.  For  each  Computer  Software 
Configuration  Item  (CSCI)  in  the  system  the  following  information  shall  be 
supplied.  The  format  for  requir«nents  stability  metric  data  is  shown  in 
Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  "That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYIY/MM/DD. 

b.  SYSTEM__NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSC1_NAME  -  The  Computer  Software  Configuration  Item  name  to  which  the 
data  applies. 

d.  R£Q_DESCRP  -  The  total  number  of  software  requirements  discrepancies 
detected  to  date.  Acceptable  values  are  in  the  range  0  through  99999. 

e.  OPEN_STAT  -  The  number  of  REQ  DESCRP  discrepancies  which  remain  open  as 
of  this  reporting  period.  Acceptable  values  are  in  the  range  0  through  99999. 

f.  CLOSED_STAT  -  The  total  number  of  R£Q_DESCRP  discrepancies  to  date  which 
are  closed  as  of  this  reporting  period.  Acceptable  values  are  in  the  range  0 
through  99999. 

g.  USER_ECP  -  The  number  of  Engineering  Change  Proposals-Software  (ECP-Ss) 
submitted  in  this  reporting  period  by  the  user  against  software  requirements. 
Acceptable  values  are  in  the  range  0  through  99999. 

h.  USER_SLOC  -  The  number  of  source  lines  of  code  affected  in  this 
reporting  period  by  approved  software  requirements  related  ECP-Ss  that  were 
submitted  by  the  user.  Source  lines  of  code  are  non-blank,  non-comment, 
executable  and  data  statements.  Acceptable  values  are  in  the  range  0  through 
9999999. 

i.  USER_MODS_AFFECT  -  The  number  of  Computer  Software  Units  (CSOs)  in  the 
CSCI  which~are  affected  in  this  reporting  period  by  approved  software 
requirements-related  ECP-Ss  submitted  by  the  user.  Acceptable  values  are  in 
the  range  0  through  99999. 

j.  DEV_ECP  -  The  number  of  ECP-Ss  submitted  in  this  reporting  period  by  the 
developer  against  software  reepairements.  Acceptable  values  are  in  the  range  0 
through  99999. 

k.  DEV_SLOC  -  The  number  of  source  lines  of  code  affected  in  this  reporting 
period  by~approved  software  requirements  related  ECP-Ss  that  were  submitted  by 
the  developer.  Source  lines  of  code  are  non-blank,  non-comment,  executable 
and  data  statements.  Acceptable  values  are  in  the  range  0  through  9999999. 
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1.  DEV_MODS_AFFECT  -  The  number  of  CSUe  in  the  CSCI  which  ere  effected  in 
this  reporting  period  by  epproved  eoftwere  requiremente-releted  ECP-Se 
submitted  by  the  developer.  Acceptable  values  ere  in  the  renge  0  through 


m.  SLOC  “  The  number  of  source  lines  of  code  in  the  CSCI.  Source  lines  of 
code  are  non-blank,  non-comment,  executable  end  data  statements.  Acceptable 
values  are  in  the  range  0  through  9999999. 

n.  NOM_SRS  REQ  -  The  number  of  Software  Requirements  Specification  (SRS) 
requirements  for  this  CSCI.  Acceptable  values  ere  in  the  renge  0  through 

99999.  V  V 


o.  NUM_SRS_REQ_ADD  -  The  number  of  SRS  requirements  added  in  this  reporting 
^99^  approved  ECP-Ss.  Acceptable  values  ere  in  the  renge  0  through 


p.  iroM_SRS_^REQ  MOD  -  The  number  of  SRS  requirements  modified  in  this 
reporting  period  due  to  epproved  ECP-Ss.  Acceptable  values  ere  in  the  renge  0 
through  99999. 

q.  NUM_SRS_^REQ  DEL  -  The  nianber  of  SRS  requirements  deleted  in  this 
reporting  period  3ue  to  approved  ECP-Ss.  Acceptable  values  ere  in  the  renge  0 
through  99999. 


TABLE  1.  Requirements  Stability  Metric  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  i/ 

DATA^DATE 

Character  2.1 

10 

SYSTEM^NAME 

Character 

20 

CSCI^NAME 

Character 

15 

REQ_DESCRP 

Numeric  2.1 

5 

0 

0PEN_STAT 

Numeric 

5 

0 

CLOSED^STAT 

Numeric 

5 

0 

USER^ECP 

Numeric 

5 

0 

USER_SLOC 

Numeric 

7 

0 

USER_MODS_AFFECT 

Numeric 

5 

0 

DEV^ECP 

Numeric 

5 

0 

DEV^SLOC 

Numeric 

7 

0 

DEV_MODS_AFFECT 

Numeric 

5 

0 

SLOC 

Numeric 

7 

0 

NUM_SRS_REQ 

Numeric 

S 

0 

NUM_SRS_REQ_ADD 

Numeric 

5 

0 

NtJM_SRS_REQ_MOD 

Numeric 

5 

0 

NUM_SRS_REQ_DEL 

Numeric 

5 

0 

1/  Decimal  - 
2.1  Character 

1/  Numeric  - 


Number  of  numerals  after  the  decimal  point. 

Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

Numerals,  decimal  point  or  negative  sign  (-). 
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PROPOSED  DRAFT.  DO  NOT  USE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 

FpmAppmftd 

OUB  No.  0704-0188 

2.  title 

SOFTWARE  DESIGN  STABILITY  METRIC  DATA  REPORT 

1.  iOE^^'IFCATlON  NUMBER 

OI-XXXX-XXXXX 

3  DESCRlPTlONrPURPOSE 

3.1  This  metric  data  is  used  to  provide  an  indication 
design  is  approaching  a  stable  state. 

I  of  whether  the  software 

{YYMMDDj 


7.  APPuCATtONtlNTERRELATlONSHlP 


6a.  OTC  applicable  6b.  GtOEP  APPLICABLE 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  design  stability  metric  data  resulting  from  the  work  task  described  by 
AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8  APPROVAL  limitation 


9s.  APPLCABLE  FORMS 


9b.  AMSC  NUMBER 


10.  PREPARATION  INSTRUCTIONS 

10.1  Reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  Format  Reo^jirement s.  The  Software  Design  Stability  Metric  Data  Report  shall 
be  comprised  of  introductory  s\immary  material  and  itemized  metric  data.  Siimmary 
material  shall  be  prepared  on  6  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
that  it  may  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

10.2.1  Tailorino  Instructions.  In  ^he  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  Hardcopy  Format..  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) •  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  ■■SubpraQraphs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  **this  (sub)  paragraph  shall.  ..**  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 

(Continued  on  Page  2) 


n.  DISTRIBUTION  STATEMENT 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  requirement 9.  The  Software  Design  Stability  Metric  Data  Report 
shall  contain  the  following  information: 

10.3.1  Title  pace.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organisation,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period (s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Desion  stability  metric  record.  For  each  Computer  Software 
Configuration  Item  (CSCI)  in  each  version  delivered  during  this  reporting 
period  the  following  information  shall  be  supplied.  The  format  for  design 
stability  metric  data  is  shown  in  Table  I. 

a.  DATA  DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  "That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYYY/MM/DD. 

b.  SYSTEM  NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  -  The  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  data  applies. 

d.  VERSION  ID  -  The  configuration  control  version  identification  of  the 
software  that"is  being  reported. 

e.  COMP  DATE  -  The  date  the  CSCI  was  assigned  VERS10N_ID.  The  date  shall 

be  expressid  as  YYYY/MM/DD.  ~ 

f.  TOT  MODULES  -  The  total  number  of  Computer  Software  Units  (CSUs)  that 
are  planned  to  comprise  the  final  delivery  of  the  CSCI.  Acceptable  values  are 
in  the  range  0  through  99999. 

g.  TOTMOD  N  DESIGN  -  The  total  number  of  CSUs  comprising  the  CSCI  delivered 
in  VERSION_ID."  Acceptable  values  are  in  the  range  0  through  99999. 

h.  FC  MOD  -  The  total  number  of  CSUs  in  which  design  related  changes  were 
made  since  the  last  delivery  of  the  CSCI.  Acceptable  values  are  in  the  range 
0  through  99999. 

i.  FA  MOD  -  The  total  number  of  CSUs  that  were  added  to  the  CSCI  since  the 
last  delivery.  Acceptable  values  are  in  the  range  0  through  99999. 

j.  FD  MOD  -  The  total  of  number  CSUs  that  were  deleted  from  the  CSCI  since 
the  last"delivery.  Acceptable  values  are  in  the  range  0  through  99999. 


C-24 


Page  2  of  3  Pages 


30  8«pt«mb«r  1992  DA  Paa  73-1,  Part  Savan  (Draft) 


TABLE  I.  Daiign  Stability  Matric  Data  Racord  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  1/ 

DATA_DATE 

Character 

1/ 

10 

SYSTEM_NAME 

Character 

20 

CSCI_NAME 

Character 

IS 

VERSION_ID 

Character 

15 

COMP_DATE 

Character 

10 

TOT__MODULES 

Numeric 

1/ 

5 

0 

TOTMOD^N_DES IGN 

Numeric 

5 

0 

FC_MOD 

Nximeric 

5 

0 

FA_MOD 

Numeric 

5 

0 

FD MOD 

Numeric 

— -a— : 

5 

0 

1/  Decimal  -  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.a.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

3/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (-). 
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PROPOSED  DRAFT.  DO  NOT  DSE  FOR  ACODISITION  PDRPOSES 


DATA  ITEM  DESCRIPTION 


FotfnApp/ov9d 
O^ylBNo.  0704<fia8 


Z  TfTLE 


software:  complexity  metric  data  report 

3.  DESCRlPTlOMPURPOSE  ~ 


1.  lOENTTiFCATlONNUMBER 
DI-XXXX-XXXXX 


3,1  This  metric  data  is  used  to  provide  an  indication  of  the  structure  of  the 
software  and  provides  a  means  to  measure,  quantify  and  evaluate  the  structure  of 
software  modules. 


^  ^  PRJMARY  RESPONSIBILFTY  (OPR) 


7.  APPllCATl0^i/INTERRElATlONSHlP 


|6a  OTIC  APPUCABLE  I  Sb.  CIOEP  APPLICABLE 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  complexity  metric  data  resulting  from  the  work  task  described  by  AR  73-1, 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


1 6.  APPROVAL  LIMriATlON 


ho.  PREPARATON  INSTRUCTIONS 


»A.  applicable  forms 


9b.  AMSC  NUMBER 


BglfilfiPCg  Ppcumentg.  The  applicable  issue  of  the  documents  cited  herein, 
including  t.heir  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

Fcrrat  Reyjirgment? .  The  Software  Complexity  Metric  Data  Report  shall  be 
comprised  of  introductory  summary  material  and  itemized  metric  data.  Summary 
material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  6  1/2  by  11  inch  paper  such 
that  it  may  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

^^•^•1  Tfliloring  Instructions*  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  .Hardcopy  Format#  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  Subpragraphs.  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  ”this  (sub) paragraph  shall...-  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 
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11.  DiSTRlBLmON  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Conf d.) 

10.3  Content  requirement a.  The  Software  Complexity  Metric  Data  Report  shall 
contain  the  following  information: 

10.3.1  Title  pace.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  nai'ne  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Complexity  metric  record.  For  each  Computer  Software  Unit  (CSU)  in 
the  system  that  has  been  added,  modified,  or  deleted  the  following  information 
shall  be  supplied.  The  format  for  complexity  metric  data  is  shown  in  Table  X. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  “That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YTYV/MM/DD. 

b.  SYSTEM_NAK£  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  -  The  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  data  applies. 

d.  CSC_NAME  -  The  Computer  Software  Component  (CSC)  name  to  which  the  data 
applies.  ~ 

e.  CSU_NAME  -  The  name  of  the  Computer  Software  Unit  (CSU)  to  which  the 
data  applies. 

f.  DELETED  -  An  indicator  that  this  CSU  has  been  deleted  from  the  CSCI. 
Acceptable  values  are  Y  (deleted)  or  N  (unit  is  part  of  the  system 
configuration) . 

g.  LANGUAGE  -  The  programming  language  the  module  is  written  in. 

h.  CYCLMTIC_COMPLX  -  The  computed  value  of  McCabe’s  cyclomatic  complexity 
for  the  CSU.  Acceptable  values  are  in  the  range  0  through  99999. 

i.  HALSTEAD_VOCAB  -  The  computed  value  of  Halstead's  vocabulary  term  for 
the  CSU.  Acceptable  values  are  in  the  range  0  through  99999. 

j.  HALSTEAD  PRO  LN  -  The  computed  value  of  Halstead's  program  length  term 
for  the  CSU.  Acceptable  values  are  in  the  range  0  through  99999. 

k.  HALSTEAD  VOL  -  The  computed  value  of  Halstead's  program  volume  term  for 
the  CSU.,  Acceptable  values  are  in  the  range  0  through  99999.99. 

l.  CTRL_PATH_CROS  -  The  total  niunber  of  occurrences  in  the  CSV  where 
control  paths  cross.  Acceptable  values  are  in  the  range  0  through  99999. 

m.  SLOC  -  The  total  number  of  source  lines  of  code  in  the  CSU.  Source 
lines  of  code  are  non-blank,  non-comment,  executable  and  data  statements. 
Acceptable  values  are  in  the  range  0  through  99999. 

n.  PCNT  CMNT  -  The  computed  percentage  of  comment  lines  in  the  CSU. 
Acceptable~value8  are  in  the  range  0  through  100. 

o.  PDL_OR_CODE  -  An  indicator  as  to  whether  the  complexity  for  this  CSU 
was  computed  on  its  Program  Design  Language  (PDL)  or  source  code 
representation.  Acceptable  values  are  PDL  or  CODE. 
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TABLE  I.  Complexity  Metric  Date  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2/ 

DATA^DATE 

Character 

2/ 

10 

SYSTEM^NAME 

Character 

20 

CSCI^NAME 

Character 

15 

CSC^NAME 

Character 

15 

CSU^NAME 

Character 

15 

DELETED 

Character 

1 

LANGUAGE 

Character 

12 

CYCLMTIC_COMPLX 

Numeric 

1/ 

5 

0 

HALSTEAD^VOCAB 

Numeric 

5 

0 

HALSTEAD_PRO_LN 

Numeric 

5 

0 

HALSTEAD^VOL 

Numeric 

8 

.  2 

CTRL_PATH_CROS 

Numeric 

5 

0 

SLOG 

Numeric 

5 

0 

PCNT^CMNT 

Numeric 

3 

0 

PDL_OR_CODE 

Character 

4 

1/  Decimal  -  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

2/  Numeric  ■>  Numerals,  decimal  point  or  negative  sign  (*■). 
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DATA  ITEM  DESCRIPTION 


9  mill 

0MB  No.  0704^18$ 


SOFTWARE  BREADTH  OF  TESTING  METRIC  DATA  REPORT 


V  lOEKrTlFCATON  NUMBER 


DI-XXXX-XXXXX 


3.  DESCRIPTION/PURPOSE 

3.1  This  metric  data  is  used  to  assess  the  degree  to  which  reqpuired  functionality 
has  been  successfully  demonstrated  as  well  as  the  amount 
of  testing  that  has  been  performed. 


4.  approval  DATE 
{YYMMDDj 


5.  OFFICE  OF  PRIMARY  RESPONSlBlLfTY  (OPR) 


6e.  OTC  APPLCABLE  i  6b.  GIOEP  APPUCABLE 


7.  APPIICATIOMNTERRELATIONSHIP 

7.1  This  DID  contains  the  format  and  content  preparation  instructions  for  software 
breadth  of  testing  metric  data  resulting  from  the  work  task  described  by  AR  73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


6.  APPROVALLIMiTATlON 


APPLCABlE  FORMS 


Sb.  AMSC  NUMBER 


1C-  PREPARATION  INSTRUCTIONS 

10.1  Reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices,  and 
revisions  shall  be  as  specified  in  the  contract. 

10.2  FcTnat  Reo^jlrements .  The  Software  Breadth  of  Testing  Metric  Data  Report  shall 
be  comprised  of  introductory  summary  material  and  itemized  metric  data.  Summary 
material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic  media.  Itemized 
data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such  that  it 
may  be  readily  compatible  with  an  electronic  database  structure.  The  structure  for 
each  type  of  data  in  the  report  is  described  in  its  corresponding  Content 
Requirement  subparagraph  below. 

10.2.1  lalloring  Instructioas*  In  ^he  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following  the 
heading  of  each  such  (sub) paragraph* 

10.2.2  Hardcopy  Format.  This  document  may  be  printed  on  one  side  or  both  sides  of 
each  page  (single-sided/double-sided) .  All  printed  pages  shall  contain  the  document 
control  number  and  publication  date  centered  at  the  top  of  the  page* 

10.2.3  Multiple  ■■PflragXAPhs ..■,flii,d....Subpra graphs >  Any  paragraph  or  subparagraph  in  this 
DID  starting  with  the  phrase  “this  (sub)  paragraph  shall...**  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 
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11.  Distribution  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Conf d.) 

10*3  Content  requirements.  The  Software  Breadth  of  Testing  Metric  Data 
Report  shall  contain  the  following  information: 

10.3.1  Title  pace.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Breadth  of  testing  metric  record.  Por  each  Computer  Software 
Configuration  Item  (CSCI)  in  the  system  the  following  information  shall  be 
supplied.  The  format  for  breadth  of  testing  metric  data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YTYT/MM/DD. 

b.  SYSTEM_NAM£  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  *■  The  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  data  applies. 

d.  NUM_SRS_REQ  -  The  total  number  of  SRS  requirements  for  the  CSCI  under 
development.  ^Acceptable  values  are  in  the  range  0  through  99999. 

e.  TESTED_SRS_REQ  -  The  total  number  of  SRS  requirements  for  the  CSCI  that 
have  been  tested  using  approved  test  cases.  Acceptable  values  are  in  the 
range  0  through  99999. 

f.  PASSED_SRS_REQ  -  The  total  number  of  SRS  requirements  for  the  CSCI  that 
have  been  successfully  demonstrated  through  testing.  Acceptable  values  are  in 
the  range  0  through  99999. 

g.  NUM_IRS_REQ  -  The  total  number  of  IRS  requirements  associated  with  the 
CSCI  under  development.  Acceptable  values  are  in  the  range  0  through  99999. 

h.  TE5TED_IRS_R£Q  -  The  total  number  of  IRS  requirements  associated  with 
the  CSCI  that  have  been  tested  using  approved  test  cases.  Acceptable  values 
are  in  the  range  0  through  99999. 

i.  PASSED_1RS_REQ  *  The  total  number  of  IRS  requirements  associated  with 
the  CSCI  that  have  been  successfully  demonstrated  through  testing.  Acceptable 
values  are  in  the  range  0  through  99999. 

j.  NUM]_UFD1  REQ  -  The  total  number  of  UFO  priority  level  one  requirements 
associated  witK  the  CSCI  under  development.  Acceptable  values  are  in  the 
range  0  through  99999. 

k.  TESTED_UFD1_REQ  -  The  total  number  of  UFD  priority  level  one 
requirements~associated  with  the  CSCI  that  have  been  tested  using  approved 
test  cases.  Acceptable  values  are  in  the  range  0  through  99999. 

l.  PASSED  UFD 1_REQ  -  The  total  number  of  UFD  priority  level  one 
requirements  associated  with  the  CSCI  that  have  been  successfully  demonstrated 
through  testing.  Acceptable  values  are  in  the  range  0  through  99999. 
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m.  NUM_UFD2_REQ  -  The  total  nxunber  of  UFD  priority  level  two  requirements 
associated  with  the  CSC!  under  development.  Acceptable  values  ere  in  the 
range  0  through  99999. 

n.  TESTED_UFD2_REQ  -  The  total  number  of  UFD  priority  level  two 
requirements^associated  with  the  CSCI  that  have  been  tested  using  approved 
test  eases.  Acceptable  values  are  in  the  range  0  through  99999. 

o.  PASSED_UFD2_REQ  -  The  total  number  of  UFD  priority  level  two 
requirement s^associated  with  the  CSCI  that  have  been  successfully  demonstrated 
through  testing.  Acceptable  values  are  in  the  range  0  through  99999. 

p.  NUM  UFD3  AEQ  -  The  total  number  of  UFD  priority  level  three  requirements 
associateH  witK  the  CSCI  under  development.  Acceptable  values  are  in  the 
range  0  through  99999. 

q.  TESTED_UFD3_A£Q  -  The  total  number  of  UFD  priority  level  three 
requirements^associated  with  the  CSCI  that  have  been  tested  using  approved 
test  cases.  Acceptable  values  are  in  the  range  0  through  99999. 

r.  PASSED_UFD3_REQ  -  The  total  number  of  UFD  priority  level  three 
requirement s~a8soeiated  with  the  CSCI  that  have  been  Successfully  demonstrated 
through  testing.  Acceptable  values  are  in  the  range  0  through  99999. 

s.  NUM_UFD4_R£Q  •  The  total  number  of  UFD  priority  level  four  requirements 
associates  with  the  CSCI  under  development.  Acceptable  values  are  in  the 
range  0  through  99999. 

t.  TESTED_UFD4__REQ  -  The  total  number  of  UFD  priority  level  four 
requirements^associated  with  the  CSCI  that  have  been  tested  using  approved 
test  cases.  Acceptable  values  are  in  the  range  0  through  99999. 

u.  PASSED_UFD4_REQ  -  The  total  number  of  UFD  priority  level  four 
requirements  associated  with  the  CSCI  that  have  been  successfully  demonstrated 
through  testing.  Acceptable  values  are  in  the  range  0  through  99999. 

V.  TEST_ID  -  The  testing  phase  or  test  event  identifier  with  which  this 
data  is  associated.  Examples  of  TEST_ID  are  FQT  (Formal  Qualification  Test), 
DT  (Government  Developmental  Test),  or  OT  (Operational  Test). 
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TABLE  I.  Breadth  of  Testing  Metric  Data  Record  Format  ) 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2/ 

DATA^DATE 

Character  2/ 

10 

SySTEM_NAME 

Character 

20 

CSCI_NAME 

Character 

IS 

NUM_SRS_REQ 

Numeric  2/ 

5 

0 

TESTED_SRS_REQ 

Numeric 

5 

0 

PASSED_SRS_REQ 

Numeric 

S 

0 

NUM_IRS_REQ 

Numeric 

5 

0 

TESTED_IRS_REQ 

Numeric 

5 

0 

PASSED_IRS_REQ 

Numeric 

5 

0 

NUM_UFD1_REQ 

Numeric 

5 

0 

TESTED_UFD1_REQ 

Numeric 

5 

0 

PASSED_UFD1_REQ 

Numeric 

5 

0 

NUM_UFD2_REQ 

Numeric 

5 

0 

TESTED_UFD2_REQ 

Numeric 

5 

0 

PASSED_UFD2_REQ 

Numeric 

5 

0 

NUM_UFD3_REQ 

Nuroeri  c 

5 

0 

TESTED_UFD3_REQ 

Numeric 

5 

0 

PASSED_UFD3_REQ 

Numeric 

5 

0 

NUM_UFD4_REQ 

Numeric 

5 

0 

TESTED_UFD4_REQ 

Numeric 

5 

0 

PASSED_UFD4_REQ 

Numeric 

5 

0 

TEST_ID 

Character 

8 

0 

1/  Decimal  ~  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 

used  in  mathematical  calculations)  or  symbols. 

2/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (-). 
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PROPOSED  DRAFT.  DO  NOT  DSE  FOR  ACQUISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


fOfmApproi^ 
OWfiAto.  0704^188 


2.  TniE 


SOFTWARE  DEPTH  OF  TESTING  METRIC  DATA  REPORT 


3.  DE  SCRIPT  lOMRURPOSE 


1.  IDENTIFICATON  NUMBER 


DI-XXXX-XXXXX 


3.1  This  metric  data  is  used  to  provide  indications  of  the  extent  and  success 
of  testing  from  the  point  of  view  of  coverage  of  possible  paths  and  conditions 
within  the  software. 


^  OFFCE  OF  PRIMARY  RESPONSlBlirTY  (OPR) 

( YYMMDul 


7.  APPUCATIONMNTERRELATIONSHIP 


6s.  DTC  APPLICABIE  I  6d.  GtOEP  APPUCABtE 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  depth  of  testing  metrrc  data  resulting  from  the  work  task  described  by  AR 
73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8.  APPROVAL  LIMITATION 


10.  PREPARATION  instructions 


9s.  APPLICABLE  FORMS 


9b.  AMSC  NUMBER 


10.1  Reference  Document s .  The  applicable  issue  of  the  documents  cited  hereinr 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  r_crr.at  Recuirement a .  The  Software  Depth  of  Testing  Metric  Data  Report  shall 
be  comprised  of  introductory  summary  material  and  itemized  metric  data.  Summary 
material  shall  be  prepared  on  6  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
that  it  may  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

10.2.1  Tailoring  Inatructions »  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  pardeopy  Tormat.  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double-sided) .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paraoraphs  and  Suboragraphs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  ''this  (sub) paragraph  shall. •«**  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 


(Continued  on  Page  2) 


11.  DISTRIBUTION  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10/  PREPARATION  INSTRUCTIONS  (Cont'd.) 

10.3  Content  reouiremente.  The  Software  Depth  of  Testing  Metric  Data  Report 
shall  contain  the  following  information: 

10.3.1  Title  paoe.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  naune  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Depth  of  testing  metric  record.  For  each  Computer  Software  Unit  (CSU) 
in  the  system  that  has  been  added,  modified,  tested,  or  deleted  since  the  last 
reporting  period  the  following  information  shall  be  supplied.  The  format  for 
depth  of  testing  metric  data  is  shown  in  Table  I.  . 

a.  DATA  DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  “That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  yYYY/MM/DD. 

b.  SYSTEM_NAHE  -  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  -  Ihe  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  data  applies. 

d.  CSC  NAME  -  The  Computer  Software  Component  (CSC)  name  to  which  the  data 
applies.  “ 

e.  CSU_NAME  -  The  Computer  Software  Unit  (CSU)  name  to  which  the  data 
applies. 

f.  DELETED  -  An  indicator  that  this  CSU  has  been  deleted  from  the  CSCI. 
Acceptable  values  are  Y  (deleted)  or  N  (unit  is  part  of  the  system 
configuration) . 

g.  PATH  sue  -  The  total  number  of  paths  in  the  CSU  that  have  been 
successfully  executed  at  least  once.  Acceptable  values  are  in  the  range  0 
through  99999. 

h.  DECPT  sue  -  The  total  number  of  decision  points  in  the  CSU  that  have 
been  successfully  tested  at  least  once  with  all  legal  classes  of  conditions 
and  one  illegal  condition.  Acceptable  values  are  in  the  range  0  through 
99999. 


i.  INPUT  sue  -  The  total  number  of  inputs  to  the  CSU  that  have  been 
successfully  tested  with  one  legal  and ‘one  illegal  entry.  Acceptable  values 
are  in  the  range  0  through  99999. 

j .  STMT  sue  -  The  total  number  of  statements  of  the  CSU  that  have  been 
successfully  tested.  Acceptable  values  are  in  the  range  0  through  99999. 

k.  PATHS  -  The  total  number  of  paths  in  the  CSU  being  reported.  Acceptable 
values  are  in  the  range  0  through  99999. 

l.  DECPTS  -  The  total  number  of  decision  points  for  the  CSU  being  reported. 
Acceptable  values  are  in  the  range  0  through  99999. 

m.  STMTS  -  The  total  number  of  executable  statements  in  the  CSU  being 
reported.  Acceptable  values  are  in  the  range  0  through  99999. 
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n.  INPT_INST  -  The  total  number  of  inputs  for  the  CSD  being  reported. 
Acceptable~valuee  are  in  the  range  0  through  99999. 

o.  TESTED_PATHS  -  The  total  number  of  paths  in  the  CSU  that  have  been 
tested  u8ing~approved  test  cases.  Acceptable  values  are  in  the  range  0 
through  99999. 

p.  TESTED_DECPTS  -  The  total  number  of  decision  points  in  the  CSU  that  have 
been  tested  using  approved  test  cases.  Acceptable  values  are  in  the  range  0 
through  99999. 

g.  TESTED_^STKNTS  -  The  total  number  of  statements  in  the  CSU  that  have  been 
tested  using'^approved  test  cases.  Acceptable  values  are  in  the  range  0 
through  99999. 

r.  TESTED_^INPT_INS  -  The  total  number  of  input  instances  to  the  CSU  that 
have  been  tested  using  approved  test  cases.  Acceptable  values  are  in  the 
range  0  through  99999. 


TABLE  I.  Depth  of  Testing  Metric  Data.  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  1/ 

DATA_DATE 

Character  2/ 

10 

SYSTEM^NAME 

Character 

20 

CSCI_NAME 

Character 

15 

CSC^NAME 

Character 

15 

CSU^NAME 

Character 

15 

DELETED 

Character 

1 

PATH^SUC 

Numeric 

5 

0 

DECPT^SUC 

Numeric 

5 

0 

INPT_SUC 

Numeric 

5 

0 

STMT_SUC 

Numeric 

5 

0 

PATHS 

Numeric 

5 

0 

DECPTS 

Numeric 

5 

0 

STMTS 

Numeric 

5 

0 

INPT_INST 

Numeric 

5 

0 

TESTED_PATHS 

Numeric 

5 

0 

tested_decpts 

NuiMric 

5 

0 

TESTED_STMNTS 

Numeric 

5 

0 

TESTED_INPT_INS 

Numeric 

5 

0 

1/  Decimal  -  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

2/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (-). 


Page  3  of  3  Pages 


C-35 


11^  I  A  flA 


30  8*pt«Bb*r  1992 


DA  PUB  73-1,  Part  Savan  (Draft) 


PROPOSED  DRAFT.  DO  NOT  USE  FOR  ACOOISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


fofmAfipr9f9d 
OMSAId.  0704^m 


SOFTWARi:  FAULT  PROFILES  METRIC  DATA  REPORT 


1.  ©ENTlFJCATKyj  NUMBER 


DI-XXXX-XXXXX 


3.  DESCRIPTIONPURPOSE 

3.1  This  metric  data  is  used  to  provide  indications  of  the  quality  of  the 
software,  as  well  as  the  ability  to  fix  known  faults. 


4.  approval  date  5.  OFFICE  OF  PRIMARY  RESPONSIBILITY  (OPR) 

(YYMMDD) 


6a.  OTC  APPLICABLE  6t>.  GIDEP  APPUCABLE 


7.  APPLICATION  INTERRELATIONSHIP 

7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  fault  profiles  metric  data  resulting  from  the  work  task  described  by  AR 
73-1. 

7.2  This  DID  may  be  applied  to  any  software  development  or  maintenance  contract. 


8.  approval  LIMITATION 


9a  APPLCABLE  FORMS 


9b.  AMSC  NUMBER 


10.  PREPARATION  INSTRUCTIONS 

10.1  reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  Fcmat  Recnjirernent s .  The  Software  Fault  Profiles  Metric  Data  Report  shall 
be  comprised  of  introductory  summary  material  and  itemized  metric  data.  Summary 
material  shall  be  prepared  on  6  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
that  It  may  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

10.2.1  Tailoring  Inst ruct ions .  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  Hardcopy  Formats  This  document  may  be  printed  on  one  side  or  both  sides 
of  each. page  (single-sided/double-sided)  .  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple  Paragraphs  and  Subpra graphs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  ”this.  (sub) paragraph  shall...**  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 
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11.  OISTRIBLITION  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 
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Block  10,  PREPARATION  INSTRUCTIONS  (Cont* d. ) 

10.3  Content  requirement 8.  -The  Software  Fault  Profilee  Metric  Data  Report 
ahall  contain  the  following  information: 

10.3.1  Title  pace.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  iecurity  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Fault  profiles  metric  record.  For  each  Computer  Software 
Configuration  Item  (CSCI)  in  the  system  the  following  information  shall  be 
supplied.  The  format  for  fault  profiles  metric  data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values  of  the  remaining  data 
elements.  ~That  is,  the  date  when  this  data  was  current.  The  date  shall  be  ' 
expressed  as  YYYY/MM/DD. 

b.  sySTEM_NAK£  *  Name  of  the  system  to  which  this  data  applies. 

c.  CSCI_NAME  -  The  Computer  Software  Configuration  Item  (CSCI)  name  to 
which  the  data  applies. 

d.  T0T_PR1_FLTS  -  The  total  number  of  priority  one  faults  detected  to  date. 
Acceptable  values  are  in  the  range  0  through  999999. 

e.  PR1_REQ_FLTS  -  The  total  ntimber  of  priority  one  requirements  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

f.  PR1_DESIGN_FLTS  -  The  total  number  of  priority  one  design  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

g.  PRl__CODE_FLTS  -  The  total  number  of  priority  one  coding  category  faults 
detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

h.  PRl  DCMNT  FITS  -  The  total  number  of  priority  one  documentation  category 
faults  detected~to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

i.  PR1_0THER_FLTS  -  The  total  number  of  priority  one  faults  to  date  not 
attributable  to~deficiencies  in  requirements,  design,  coding  or  documentation 
categories.  Acceptable  values  are  in  the  range  0  through  999999. 

j.  TOT  PRl  CLSDFLTS  -  The  total  number  of  priority  one  faults  that  have 
been  closed/resolved  to  date.  Acceptable  values  are  in  the  range  0  through 
999999. 


k.  TOT  PR2  FITS  -  The  total  number  of  priority  two  faults  detected  to  date. 
Acceptable  values  are  in  the  range  0  through  999999. 

l.  PR2  REQ_FLTS  -  The  total  number  of  priority  two  requirements  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

m.  PR2  DESIGN  FITS  -  The  total  number  of  priority  two  design  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

n.  PR2  CODE_FLTS  -  The  total  number  of  priority  two  coding  category  faults 
detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

o.  PR2_DCMNT_FLTS  -  The  total  number  of  priority  two  documentation  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 
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P*  PR2_OTHER__FLTS  -  The  total  number  of  priority  two  faults  to  date  not 
attributable  to  deficiencies  in  requirements,  design,  coding  or  documentation 
categories.  Acceptable  values  are  in  the  range  0  through  999999. 

q.  TOT_PR2_CLSDFLTS  -  The  total  number  of  priority  two  faults  that  have 
been  closed/resolved  to  date.  Acceptable  values  are  in  the  range  0  through 
999999. 

r .  TOT_PR3_FLTS  -  The  total  number  of  priority  three  faults  detected  to 
date.  Acceptable  values  are  in  the  range  0  through  999999. 

*•  PR3_REQ_FLTS  -  The  total  number  of  priority  three  requirements  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

t.  PR3_DES1GN_FLTS  -  The  total  number  of  priority  three  design  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

u.  PR3_C0DE_FLTS  -  The  total  number  of  priority  three  coding  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

V.  PR3_DCMNT__FLTS  -  The  total  number  of  priority  three  documentation 
category  faults  detected  to  date.  Acceptable  values  are  in  the  range  0 
through  999999. 

w.  PR3_OTHER_FLTS  -  The  total  number  of  priority  three  faults  to  date  not 
attributable  to  deficiencies  in  requirements,  design,  coding  or  documentation 
categories.  Acceptable  values  are  in  the  range  0  through  999999. 

X.  T0T_PR3_CLSDFLTS  -  The  total  number  of  priority  three  faults  that  have 
been  closed/resolved  to  date.  Acceptable  values  are  in  the  range  0  through 
999999.  ^ 

y.  TOT_PR4_FLTS  -  The  total  number  of  priority  four  faults  detected  to 
date.  Acceptable  values  are  in  the  range  0  through  999999. 

PR4_REQ_FLTS  -  The  total  number  of  priority  four  requirements  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

PR4_DESIGN_FLTS  “  The  total  number  of  priority  four  design  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

bb.  PR4_C0DE_FLTS  -  The  total  number  of  priority  four  coding  category  faults 
detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

cc.  PR4  DCMNT_FLTS  -  The  total  number  of  priority  four  documentation 
category  Taults~detected  to  date.  Acceptable  values  are  in  the  range  0 
through  999999. 

dd.  PR4  OTHER_FLTS  -  The  total  number  of  priority  four  faults  to  date  not 
attributable  to“deficiencies  in  requirements,  design,  coding  or  documentation 
categories.  Acceptable  values  are  in  the  range  0  through  999999. 

ee.  T0T_PR4_CLSDFLTS  -  The  total  number  of  priority  four  faults  that  have 
been  closed/resolved  to  date.  Acceptable  values  are  in  the  range  0  through 
999999.  ^ 

ff.  TOT_PR5_FLTS  —  The  total  number  of  priority  five  faults  detected  to 
date.  Acceptable  values  are  in  the  range  0  through  999999. 

gg.  PR5_REQ_FLTS  —  The  total  number  of  priority  five  requirements  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

hh.  PR5_DESIGN_FLTS  ~  The  total  number  of  priority  five  design  category 
faults  detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 
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ii.  PR5_CODE_rLTS  -  The  total  number  of  priority  five  coding  category  faults 
detected  to  date.  Acceptable  values  are  in  the  range  0  through  999999. 

jj.  PR5  DCMNT_FLTS  -  The  total  number  of  priority  five  documentation 
category  zaults  detected  to  date.  Acceptable  values  are  in  the  ranae  0 
through  999999.  ^ 

kk.  PRS_OTHER_FLTS  -  The  total  number  of  priority  five  faults  to  date  not 
attributable  to  deficiencies  in  requirements,  design,  coding  or  documentation 
categories.  Acceptable  values  are  in  the  range  0  through  999999. 

11.  TOT_PR5_CLSDFLTS  -  The  total  number  of  priority  five  faults  that  have 
been  closed/resolved  to  date.  Acceptable  values  are  in  the  range  0  through 
999999. 

mm.  AVG0PEN_AGE_PR1  -  The  average  number  of  days  a  currently  open  priority 
one  fault  has  remained  open.  Acceptable  values  are  in  the  range  0  through 
99999. 


nn.  AVGOPEN_AGE_PR2  -  The  average  number  of  days  a  currently  open  priority 
two  fault  has  remained  open.  Acceptable  values  are  in  the  range  0  through 
99999. 


oo.  AVGOPEN_AGE_PR3  -  The  average  hiimber  of  days  a  currently  open  priority 
three  fault  has  remained  open.  Acceptable  values  are  in  the  range  0  through 
99999. 


pp.  AVGOPEN_AGE_PR4  -  The  average  number  of  days  a  currently  open  priority 
four  fault  has  remained  open.  Acceptable  values  are  in  the  range  0  through 
99999. 


qg.  AVGOPEN_AGE_PR5  -  The  average  number  of  days  a  currently  open  priority 
five  fault  has  remained  open.  Acceptable  values  are  in  the  range  0  through 
99999. 


rr.  AVGCL0SE_AGE_PR1  -  The  average  number  of  days  it  took  to  close  a 
priority  one  faultT  Acceptable  values  are  in  the  range  0  through  99999. 

ss.  AVGCLOSE_AGE_PR2  -  The  average  number  of  days  it  took  to  close  a 
priority  two  faultT  Acceptable  values  are  in  the  range  0  through  99999. 

tt.  AVGCLOSE_AGE  PR3  —  The  average  number  of  days  it  took  to  close  a 
priority  three  fault.  Acceptable  values  are  in  the  range  0  through  99999. 

uu.  AVGCLOSE  AGE_PR4  -  The  average  number  of  days  it  took  to  close  a 
priority  four  fault.  Acceptable  values  are  in  the  range  0  through  99999. 

vv.  AVGCLOSE  AGE_PR5  -  The  average  number  of  days  it  took  to  close  a 
priority  five  fault.  Acceptable  values  are  in  the  range  0  through  99999. 

ww.  AVG_AGE_PR1  -  The  average  age  of  a  priority  one  fault  (both  open  and 
closed).  .Acceptable  values  are  days  in  the  range  0  through  99999. 

XX.  AVG_AGE_PR2  -  The  average  age  of  a  priority  two  fault  (both  open  and 
closed).  Acceptable  values  are  days  in  the  range  0  through  99999. 

yy*  AVG_AGE_PR3  -  The  average  age  of  a  priority  three  fault  (both  open  and 
closed).  Acceptable  values  are  days  in  the  range  0  through  99999. 

»*.  AVG_AGE_PR4  -  The  average  age  of  a  priority  four  fault  (both  open  and 
closed).  Acceptable  values  are  days  in  the  range  0  through  99999. 

aaa.  AVG_AGE_PR5  -  The  average  age  of  a  priority  five  fault  (both  open  and 
closed).  Acceptable  values  are  days  in  the  range  0  through  99999. 
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TABLE  I.  Fault  Profiles  Data  Record  Format 


- 

Maximum 

Data  Element  Name 

Data  Type 

Length 

Decimal  1/ 

DATA_DATE 

Character  2/ 

10 

SYSTEM^NAME 

Character 

20 

CSCI^NAME 

Character 

15 

T0T_PR1_FLTS 

Numeric  2/ 

6 

0 

PR1_REQ_FLTS 

Numeric 

6 

0 

PR1_DESIGN_FLTS 

Numeric 

6 

0 

PR1_C0DE_FLTS 

Numeric 

6 

0 

PR1_DCMNT_FLTS 

Numeric 

6 

0 

PR1_0THER_FLTS 

Numeric 

6 

0 

T0T_PR1_CLSDFLTS 

Numeric 

6 

0 

T0T_PR2_FLTS 

Numeric 

6 

0 

PR2__REQ_FLTS 

Numeric 

6 

0 

PR2_DESIGN_FLTS 

Numeric 

6 

0 

PR2_C0DE_FLTS 

Numeric 

6 

0 

PR2__DCMNT_FLTS 

Numeric 

6 

0 

PR2_OTHER_FLTS 

Numeric 

6 

0 

T0T_PR2_CLSDFLTS 

Numeric 

6 

0 

T0T_PR3_FLTS 

Numeric 

6 

0 

PR3_REQ_FLTS 

Numeric 

6 

0 

PR3_DESIGN_FLTS 

Numeric 

6 

0 

PR3_C0DE_FLTS 

Numeric 

6 

0 

PR3_DCMNT_FLTS 

Numeric 

6 

0 

PR3_0THER_FLTS 

Numeric 

6 

0 

T0T_PR3_CLSDFLTS 

Numeric 

6 

0 

T0T_PR4_FLTS 

Numeric 

6 

0 

PR4_REQ_FLTS 

Numeric 

6 

0 

PR4_DESIGN_FLTS 

Numeric 

6 

0 

PR4_C0DE_FLTS 

Numeric 

6 

0 

PR4_DCMNT_FLTS 

Numeric 

6 

0 

PR4_OTHER_FLTS 

Numeric 

6 

0 

T0T_PR4_CLSDFLTS 

Numeric 

6 

0 

T0T_PR5_FLTS 

Numeric 

6 

0 

PR5_REQ_FLTS 

Numeric 

6 

0 

PR5_DESI6N_FLTS 

Numeric 

6 

0 

PR5_C0DE_FLTS 

Numeric 

6 

0 

PR5_DCMNT_FLTS 

Numeric 

6 

0 

PR5_OTHER_FLTS 

Numeric 

6 

0 

T0T_PR5_CLSDFLTS 

Numeric 

6 

0 

AVG0PEN_AGE_PR1 

Numeric 

5 

0 

AVGOPEN_AGE_PR2 

Numeric 

5 

0 

AVGOPEN_AGE_PR3 

Numeric 

5 

0 

AVGOPEN_AGE_PR4 

Numeric 

5 

0 

AVGOPEN_AGE_PR5 

Numeric 

5 

0 
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Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  1/ 

AVGCL0SE_AGE_PR1 

Numeric 

5 

0 

AVGCLOSE_AGE_PR2 

Numeric 

5 

0 

AVGCLOSE__AGE_PR3 

Numeric 

5 

0 

AVGCLOSE_AGE_PR4. 

Numeric 

5 

0 

AVGCL0SE_AGE_PR5 

Numeric 

5 

0 

AVG_AGE_PR1 

Numeric 

5 

0 

AVG_AGE_PR2 

Numeric 

5 

0 

AVG_AGE_PR3 

Numeric 

5 

0 

AVG_AGE_PR4 

Numeric 

5 

0 

AVG_AGE_PR5 

Numeric 

5 

0 

1/  Decimal  - 
2.1  Character 

2/  Numeric  - 


Number  of  numeral*  after  the  decimal  point. 

Alphabetic  character*,  numeric  character*  (l.a.  cannot  be 
u*ed  in  mathematical  calculation*)  or  *ymbol*. 

Numerals,  decimal  point  or  negative  sign  (-). 
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PROPOSED  DRAFT.  DO  NOT  OSE  FOR  ACODISITION  PURPOSES 


DATA  ITEM  DESCRIPTION 


FOffTt  AppfV¥0d 
OMB  No.  0704^188 


2  TITLE 


SOFTWARE  RELIABILITY  METRIC  DATA  REPORT 


1.  lOENTTlFlCATON  NUMBER 


DI-XXXX-XXXXX 


3.  DESCRiPTlONPURPOSE 


3.1  This  metric  data  is  used  to  indicate  rates  at  which  faults  are  being  reduced 
and  thus  reliability  increased. 


4.  approvaldate 

{YYMMDDi 


1  OFFICE  OF  PRIMARY  RESPONSiBlLrTY  (OPR) 


6e.  OTC  APPLICABLE  I  6b.  GIOEP  APPLICABLE 


7.  APPLICATION  interrelationship 


7.1  This  DID  contains  the  format  and  content  preparation  instructions  for 
software  reliability  metric  data  resulting  from  the  work  task  described  by  AR  73-1 . 

7.2  This  DID  may  be  applied  in  any  software  development  or  maintenance  contract. 


8.  APPROVAL  LIMITATION 


te.  applicable  forms 


9b.  AMSC  NUMBER 


1C,  PREPARATION  INSTRUCTIONS 


10.1  Reference  Documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices, 
and  revisions  shall  be  as  specified  in  the  contract. 

10.2  Fcrrr.at  Pecuirement s .  The  Software  Reliability  Metric  Data  Report  shall  be 
comprised  of  introductory  summary  material  and  itemired  metric  data.  Summary 
material  shall  be  prepared  on  8  1/2  by  11  inch  paper  or  electronic  media. 

Itemized  data  shall  be  supplied  on  electronic  media  or  8  1/2  by  11  inch  paper  such 
that  It  may  be  readily  compatible  with  an  electronic  database  structure.  The 
structure  for  each  type  of  data  in  the  report  is  described  in  its  corresponding 
Content  Requirement  subparagraph  below. 

10.2.1  Tailorino  Instructions.  In  the  event  that  a  paragraph  or  subparagraph  has 
been  tailored  out,  a  statement  to  that  effect  shall  be  added  directly  following 
the  heading  of  each  such  (sub) paragraph. 

10.2.2  Hardcopy  Format.  This  document  may  be  printed  on  one  side  or  both  sides 
of  each  page  (single-sided/double*sided) •  All  printed  pages  shall  contain  the 
document  control  number  and  publication  date  centered  at  the  top  of  the  page. 

10.2.3  Multiple^aragraphs  and  Subpraoraohs .  Any  paragraph  or  subparagraph  in 
this  DID  starting  with  the  phrase  **this  (sub) paragraph  shall...**  may  be  written  as 
multiple  paragraphs  or  subparagraphs  to  enhance  readability. 


(Continued  on  Page  2) 


11.  DISTRIBLTTION  STATEMENT 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release,  distribution  is  unlimited. 


DO  Forniie64,  JUN  66 


Previous  eations  are  obsolete 
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Block  10,  PREPARATION  INSTRUCTIONS  (Confd.) 

10.3  Content  requirement b.  The  Software  Reliability  Metric  Data  Report  shall 
contain  the  following  information: 

10.3.1  Title  page.  This  page  shall  contain  the  name  and  any  abbreviation  or 
acronym  of  the  system,  name  of  sponsoring  or  issuing  organization,  document 
type  name,  contract  number,  name  of  contractor,  and  publication  date,  as  well 
as  any  necessary  security  markings  or  other  restrictions  on  the  handling  of 
the  document. 

10.3.2  Summary  of  activity.  This  paragraph  shall  specify  the  start  and  end 
dates  of  the  reporting  period(s)  and  number  of  data  records  supplied  for  the 
data  in  paragraph  10.3.3.  Significant  or  unusual  circumstances  affecting  the 
preparation  of  the  data  records,  if  any,  shall  be  described. 

10.3.3  Reliability  metric  record.  For  each  system  test  event  for  which 
system/software  reliability  data  is  measured  the  following  information  shall 
be  supplied.  The  format  for  reliability  metric  data  is  shown  in  Table  I. 

a.  DATA_DATE  -  The  date  associated  with  the  values. of  the  remaining  data 
elements.  That  is,  the  date  when  this  data  was  current.  The  date  shall  be 
expressed  as  YYYY/MH/OD. 

b.  SYSTEM_NAME  -  Name  of  the  system  to  which  this  data  applies. 

c.  TEST^ID  -  The  testing  phase  or  test  event  identifier  with  which  this 
data  is  associated.  Examples  of  TEST_ID  are  RGT  (Reliability  Growth  Test),  DT 
(Government  Developmental  Test)  Phase~I  and  OT  (Operational  Test). 

d.  KEASURE0_FAIL_RATE  -  The  computed  failure  rate  of  the  software  as 
measured  in  testing~expressed  in  failures  per  month.  Acceptable  values  are  in 
the  range  0  through  99999.99. 

e.  PROJECTED_FAIL_RATE  -  The  projected  failure  rate  of  the  software  for  the 
reporting  period  as  calculated  by  the  reliability  analysis  model  expressed  in 
failures  per  month.  Acceptable  values  are  in  the  range  0  through  99999.99. 

f.  RELY  MODEL  ■>  The  name  of  the  analytical  model  used  to  calculate  the 
PROJECTED_FAILURE_RATE . 

g.  FAIL_RATE_OBJECTIVE  -  The  desired  goal  of  the  acceptable  number  of 
software  failures  per  month.  Acceptable  values  are  in  the  range  0  through 
99999.99. 

h.  SYS_R£Q_KTBF  -  The  required/specified  value  of  mean  time  between  mission 
failures  caused  by  system  hardware  or  software  against  which  the  value 
measured  during  the  test  is  compared  for  compliance.  Acceptable  values  are  a 
number  of  hours  in  the  range  0  through  99999.99. 

i.  PBST_SYS  HTBF  -  The  computed  point  estimate  of  mean  time  between  mission 
failures  caused  by  system  hardware  or  software  as  measured  during  the  test 
event  (field  TEST  ID).  Acceptable  values  are  a  number  of  hours  in  the  range  0 
through  99999.99.” 

j.  LCB_SYS_MTBF  -  The  calculated  80%  lower  confidence  bound  value  of  mean 
time  between  mission  failures  caused  by  system  hardware  or  software  as 
measured  during  the  test  event.  Acceptable  values  are  a  number  of  hours  in 
the  range  0  through  99999.99 

k.  PEST_SH_MTBF  -  The  computed  point  estimate  of  mean  time  between  mission 
failures  caused  by  software  as  measured  during  the  test  event.  Acceptable 
values  are  a  number  of  hours  in  the  range  0  through  99999.99. 
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1«  LCB_SW_MTBF  —  The  c&lculfttcd  80%  lower  confidence  bound  velue  of  Been 
time  between  mission  failures  caused  by  software  as  measured  during  the  test 

QQooQ*  Acceptable  values  are  a  number  of  hours  in  the  range  0  through 
99999 • 99 • 


m.  REQ_MEAN_RESTOR  -  The  required/epecified  value  of  mean  time  to  restore 
the  system  to  operational  status.  Acceptable  values  are  a  number  of  hours  in 
the  range  0  through  99999.99. 

n.  REQ_MEDN__RESTOR  -  The  required/specified  value  of  median  time  to  restore 
the  system  to  operational  status.  Acceptable  values  are  a  number  of  hours  in 
the  range  0  through  99999.99. 

®*  .  ®^Q_**AX95_REST0R  -  The  required/specified  maximum  95th  percentile  value 

ox  txme  to  restore  the  system  to  operational  status.  Acceptable  values  are  a 
number  of  hours  in  the  range  0  through  99999.99. 

p.  MAN  RESTOR_SYS  -  The  computed  mean  time  to  restore  the  system  to 
©^rational  condition.  Acceptable  values  are  a  number  of  hours  in  the  range  0 
through  99999.99. 

q.  ^DN  RESTOR_sys  -  The  computed  median  time  to  restore  the  system  to 

operational  condition.  Acceptable  values  are  a  number  of  hours  in  the  range  0 
through  99999.99.  ’ 

r.  KAX95_^RESTOR_Sys  -  The  computed  maximum  95th  percentile  value  of  time  to 
restore  the  system  to  operational  condition.  Acceptable  values  are  a  number 
of  hours  in  the  range  0  to  99999.99. 


TABLE  I.  Reliability  Metric  Data  Record  Format 


Data  Element  Name 

Data  Type 

Maximum 

Length 

Decimal  2,/ 

DATA_DATE 

Character  2/ 

10 

SySTEM_NAME 

Character 

20 

TEST_ID 

Character 

8 

MEASUR£D_FAIL_RATE 

Numeric  2/ 

8 

2 

PROJECTED_FAIL_RATE 

Numeric 

8 

2 

RELy_MODEL 

Character 

15 

FAIL_RATE_OBJECTIVE 

Numeric 

8 

2 

SyS_REQ_MTBF 

Numeric 

8 

2 

PEST_SyS_MTBF 

Numeric 

8 

2 

LCB_Sys_MTBF 

Numeric 

8 

2 

PEST__SW_MTBF 

Numeric 

8 

2 

LCB_SW__MTBF 

Numeric 

8 

2 

REQ_MEAN_RESTOR 

Numeric 

8 

2 

REQ_MEDN_RESTOR 

Numeric 

8 

2 

REQ_MAX9  5_RESTOR 

Numeric 

8 

2 

MEAN_RESTOR_Sys 

Numeric 

8 

2 

j  MEDN_RESTOR_SyS 

Numeric 

8 

2 

MAX9 5_RES TOR_Sy S 

Numeric 

8 

2 

1/  Decimal  -  Number  of  numerals  after  the  decimal  point. 

2/  Character  -  Alphabetic  characters,  numeric  characters  (i.e.  cannot  be 
used  in  mathematical  calculations)  or  symbols. 

1/  Numeric  -  Numerals,  decimal  point  or  negative  sign  (-) 
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Appandiz  D 

Automatad  Toola  and  Matries  Data  Cellaetien 
D-l.  Introduction 

This  appendix  documents  a  preliminary  survey  of 
automated  .tools  to  support  collection  of  the  specific 
data  elements  required  for  metrics  reporting  outlined 
in  this  part  of  DA  Pamphlet  73-1  .  It  provides  program 
managers  and  program  executive  officers  with  a  listing 
of  potential  tools  which  can  aid  in  the  collection  and 
reporting  of  software  metrics. 

D-2.  Survey  objective 

a.  In  order  for  a  tool  to  be  included  in  this 
appendix  the  following  criteria  must  be  met  to  a 
significant  degree: 

(1)  A  sufficient  number  of  data  elements  that 
comprise  at  least  one  metric  are  collected  by  the  tool. 

(2)  The  media  options  provided  by  the  tool's 
output  are  sufficient  to  support  the  effective  transfer 
of  information  to  the  metrics  database. 

b.  The  term  "sufficient"  in  the  case  of  the  first 
criterion  varies  according  to  each  metric.  The  media 
options  necessary  also  vary  in  this  way.  The  reason 
for  so  few  criteria  is  primarily  the  need  to  provide 
tool  support  for  all  possible  platforms  and  languages 
which  may  be  used  in  systems  development. 

D-3 .  Survey  methodology 

a.  Existing  repositories  of  automated  software  tool 
studies  were  investigated  for  likely  candidates  for 
metrics  collection.  These  repositories  included: 

(1)  Software  Technology  Support  Center  (STSC) . 

(2)  Strategic  Defense  Initiative  Office  (SDIO) . 

(3)  Data  &  Analysis  Center  for  Software  (DACS) . 

(4)  General  Services  Administration  (GSA) . 

(5)  National  Institute  of  Science  and  Technology 
(NIST) . 

(6)  Ada  Information  Clearinghouse  (AdalC) . 

b.  Host  of  the  studies  above  provide  product 
overviews  of  tools  that  meet  very  general  criteria  in 
broad  categories.  Examples  are  requirements  analysis 
tools  and  testing  tools.  In  many  cases  detailed 
product  information  such  as  specific  data  inputs, 
output  data  items,  and  types  of  output  media  were  not 
identified.  These  reports  did,  however,  provide  a  list 
of  vendors  that  could  be  contacted  for  additional 
information. 

c.  For  each  of  the  twelve  metrics  a  vendor  survey 
form  was  created.  The  survey  form  requested  overall 
information  such  as  media  options  for  output,  platforms 
and  languages  supported  by  the  tool,  and  description  of 
the  tool's  capabilities.  The  specific  metric  data 
elements  of  interest  as  identified  in  the  data 
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requirements  of  Chapter  17,  Software  TiE  Metrics,  and 
Appendix  D,  Metrics  Data  Collection,  of  this  part  of  DA 
Pamphlet  73-1  were  included  on  the  form.  Once  it  was 
determined  which  metric (s)  a  particular  vendor's  tool 
was  designed  to  collect,  the  appropriate  survey  form(s} 
was  sent  to  the  vendor  for  completion.  A  brief 
description  of  the  Army  Software  Test  and  Evaluation 
Panel  (STEP)  effort  and  purpose  of  the  survey  was 
included.  The  responses  to  the  survey  forms  and 
telephone  follow-up  are  the  basis  of  the  products 
identified  in  this  appendix. 

d.  A  complete  list  of  products  considered  in  the 
vendor  survey  is  supplied  in  paragraph  D-20. 

D-4 .  Constraints 

The  appendix  does  not  represent  an  endorsement  of  a 
tool's  capability  to  provide  the  required  data 
elements.  Although  designed  to  ensure  that  the  tools 
polled  actually  addressed  metrics  collection  and 
reporting  requirements,  the  initial  vendor  survey  is 
not  sufficient  to  certify  that  each  tool  listed  will 
perform  as  required.  The  majority  of  the  surveys  were 
answered  by  the  tool  vendors  themselves  and  depend  upon 
each  vendors'  successful  evaluation  of  their  tool's 
capability  in  collecting  and  reporting  the  data 
elements  listed  on  the  survey  form.  It  is  expected 
upon  publication  of  this  part  of  DA  Pamphlet  73-1  that 
tool  vendors  will  request  a  copy  of  this  appendix  in 
order  to  satisfy  real  or  perceived  discrepancies. 

D-5.  Tool  certification 

It  will  be  necessary  to  investigate  whether  the 
representation  of  a  tool's  capabilities  accurately 
depicts  its  performance.  This  can  be  accomplished 
through  a  demonstration  provided  by  the  vendor  using 
real  data.  Alternatively,  in  cases  where  a  new  release 
of  the  tool  is  imminent,  the  Army  can  provide  the 
vendor  with  beta  test  support.  Furthermore,  a 
certification  process  can  provide  an  opportunity  to 
investigate  other  factors  which  could  influence  the 
selection  and  acquisition  of  a  tool  including:  cost, 
usability,  technical  support,  and  additional 
functionality.  The  publication  of  this  appendix 
precedes  the  initiation  of  a  tool  certification 
process. 

D-6.  Overall  tool  support  for  metrics 
This  paragraph  summarizes  the  capabilities  required  of 
specific  tools  for  metrics  data  collection  and 
automated  tool  availability.  Some  metrics  have  been 
grouped  together  for  discussion  because  they  deal  with 
similar  or  closely  linked  data  requirements. 

a.  Cost  and  schedule.  There  are  a  wide  range  of 
tools  one  can  choose  from  that  will  more  than  fulfill 
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the  data  requirenents  for  the  cost  and  schedule  metric 
(as  well  as  the  optional  manpower  metric) .  Many  of  the 
tools  can  construct  a  work  breakdown  structure  (WBS) 
defined  by  the  user,  a  schedule  with  user  defined 
milestones,  and  the  integration  of  costs  with  selected 
WBS  elements.  Planned  and  actual  costs  and  schedule 
progress  are  also  supported  for  the  vast  majority  of 
these  tools.  It  is  also  likely  that  existing  cost 
accounting  systems  within  a  developer's  organization 
will  collect  the  necessary  data.  This  appendix 
includes  some  examples  of  tools  that  can  capture  the 
required  data  elements  for  cost  and  schedule,  and  also 
provide  a  number  of  other  capabilities. 

b.  Computer  resource  utilization.  Many  computer 
systems  have  an  accounting  function  that  provides 
internal  statistics  on  system  performance  and  capacity 
management.  The  following  represent  capacity  planning 
and  system  performance  measurements  typically  available 
from  a  computer's  own  internal  accounting  system: 

(1)  CPU  clock  time.  Measures  the  amount  of  time 
consumed  by  various  CPU  processes. 

(2)  Channel  activity.  Measures  the  number  of 
requests  by  the  CPU  to  transfer  information  to 
peripheral  devices. 

(3)  Communications  activity.  Measures  the 
amount  of  time  required  to  transmit  data. 

(4)  Percent  CPU  utilization.  Records  the 
proportion  of  time  the  CPU  was  active  and  providing 
resources  to  a  user  request  and  unavailable  for  use  by 
other  users. 

(5)  DASD  storage  available.  The  direct  access 
storage  device  statistic  measures  the  amount  of 
auxiliary  storage  addressable  and  available  to  the  CPU. 

(6)  Memory.  Measures  the  amount  of  memory  (main 
storage)  addressable  directly  by  the  CPU. 

(7)  I/O  utilization  data.  Measures  amount  of 
time  for  information  transfer  to  and  from  input/output 
devices. 

These  measurements  are  typically  expressed  in  terms 
which  may  require  conversion  before  the  measurements 
can  be  expressed  in  a  format  suitable  for  metrics 
reporting  as  defined  in  this  part  of  DA  Pamphlet  73-1. 
Additional  tools,  however,  can  provide  the  necessary 
formats  for  CRU  metric  expression.  These  tools  are 
designed  and  developed  for  the  performance  measurement 
of  a  very  specific  series  of  computer  systems.  These 
products  reduce  the  time  and  effort  required  to  manage 
current  and  future  system  performance.  They  provide 
expansive  functionality  and  an  easy  user  interface  to  a 
widfs  range  of  sophisticated  features.  Difficulties  in 
measuring  system  performance  arise  when  one  attempts  to 
accommodate  a  variety  of  multiprocessor  architectures. 
The  configurations  can  range  from  highly  parallel 
structures,  where  processing  elements  are  tightly 
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coupled  and  used  to  solve  only  one  problem  at  a  time, 
to  loosely  coupled  systems  where  each  processor  is 
working  on  a  different  problem.  Many  of  the  previously 
listed  measurements  for  single  processors  can 
accommodate  a  multiprocessor  environment  if  the  number 
of  active  .processors  can  be  included  in  the  equations 
and  the  difference  in  processor  capabilities  can  be 
accounted  for  in  order  to  provide  a  common  baseline  of 
analysis.  Performance  management  tools  can  also 
provide  support  for  these  multiprocessor  environments. 

c.  Software  Engineering  Environment  (SEE) .  It  is 
questionable  whether  automating  the  forms  that  comprise 
the  Software  Engineering  Institute's  (SEI)  capability 
assessment  will  enhance  the  collection  process  of  the 
SEE  metric.  An  assessment  of  an  organization's 
software  development  process  requires  an  in-depth 
investigation  of  policies  and  practices;  automating 
even  a  portion  of  the  assessment  process  could 
compromise  the  high  level  of  human  interaction 
necessary  to  successfully  conduct  an  assessment.  The 
SEI  has  not  developed  an  automated  version  of  the 
assessment  process  and  no  publicly  available  tool 
exists.  Some  vendors  have  developed  automated  forms 
for  a  developer's  own  use  in  assessing  an 
organization's  capability,  but  these  are  not  intended 
to  replace  the  actual  assessment  process.  At  present 
the  only  data  required  for  the  SEE  metric  from  the 
actual  assessment  process  are  the  overall  maturity 
level  of  the  developer's  organization  and  the  date  the 
maturity  level  was  assigned.  The  collection  of  these 
data  do  not  require  automated  support. 

d.  Requirements  traceability.  Automatic 
calculation  of  forward  and  backward  tracing  between 
requirements  documents,  design  and  code  would  be  ideal 
for  the  automated  support  of  this  metric. 

Unfortunately,  many  of  the  tools  capable  of 
requirements  traceability  are  dependent  upon  one  or 
more  specific  methodologies  for  requirements  analysis 
and  design.  These  methodologies  are  expressed  through 
formal  notation  such  as  data  flow  diagrams  or  entity 
relationship  diagrams.  It  is  typically  these  notations 
that  are  used  by  the  tools  to  successfully  conduct  a 
trace  of  requirements  from  the  development  of  the 
requirements  to  the  development  of  test  cases.  The 
requirements  traceability  metric  calls  for  documented 
requirements  meaning  that  .in  order  for  an  automated 
tool  to  capture  and  trace  requirements  through  the  life 
cycle  requires  the  identification  and  parsing  of 
requirements  statements  from  text.  Increasingly, 
vendors  are  developing  CASE  tools  which  address  the 
entire  software  life  cycle.  The  requirements 
traceability  metric  requires  the  kind  of  support  that, 
at  least  for  the  present,  only  a  CASE  tool  can  provide. 
Very  few  of  these  tools  have  been  developed  in  support 
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of  the  stringent,  text-based/  tracing  requirements  of  a 
system  developed  for  the  Department  of  Defense. 

e.  Requirements  stability,  design  stability  and 
breadth  of  testing.  These  metrics  are  not  as  data 
intensive  as  the  complexity  and  depth  of  testing 
metrics  aiid  do  not  necessarily  require  automated 
support.  Two  categories  of  tools  provide  some  type  of 
support  for  these  metrics. 

(1)  One  of  the  benefits  of  finding  a  tool  that 
supports  requirements  traceability  is  that  it  will 
typically  support  the  other  metrics  that  directly 
concern  changes  in  the  requirements  and  design  of  the 
product  over  the  entire  life  cycle.  The  two  stability 
metrics  deal  with  these  changes.  The  unique  data 
elements  that  comprise  the  breadth  of  testing  metric 
merely  comprise  an  accumulated  total  of  requirements 
drawn  from  the  requirements  traceability  metric  that 
have  been  tested  (and  either  passed  or  failed)  and 
those  successfully  tested.  Therefore,  a  tobl  that  can 
collect  the  data  elements  that  comprise  the 
requirements  traceability  metric  is  also  likely  to 
capture  (or  can  be  adapted  to  capture)  the  data 
elements  that  comprise  the  breadth  of  testing  metric  as 
well. 

(2)  Configuration  management  tools  may  address 
design  stability  and  some  fault  tracking  systems  will 
indirectly  collect  data  for  requirements  stability;  but 
requirements  stability,  design  stability  and  breadth  of 
testing  do  not  have  tools  specifically  designed  for 
collection  and  reporting  of  the  data  elements  described 
in  this  part  of  DA  Pamphlet  73-1.  Configuration 
management  tools  do  not  provide  the  kind  of  output 
necessary  to  support  an  effective  transfer  of  data  to  a 
metrics  database. 

f.  Complexity  and  depth  of  testing.  Automated 
support  for  the  data  elements  comprising  the  complexity 
depth  of  testing  metrics  is  necessary  due  to  the 
computational  nature  and  data  volume  associated  with 
these  metrics.  Many  commercial  vendors  have  developed 
tools  that  calculate  the  Halstead  metrics  and  McCabe's 
cyclomatic  complexity.  Some  compilers  also  compute 
complexity,  but  they  are  not  included  in  this  analysis. 
To.als  that  collect  instances  of  control  flow  crossings 
within  a  CSU  are  much  less  common.  Commercial  vendors 
supply  automated  tools  that  support  many  of  the  data 
elements  comprising  the  depth  of  testing  metric  under 
the  label  of  coverage  analysis.  While  there  are  many 
tools  which  collect  some  aspect  of  complexity  and 
coverage  analysis,  there  are  few  tools  which  collect 
all  the  data  elements  required  for  either  metric  and 
even  fewer  that  collect  all  the  data  elements  required 
for  both.  There  is  a  trend  towards  the  development  of 
"tool  suites"  which,  if  used  as  a  complete  package, 
will  provide  many  of  the  data  elements  required  for 
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complexity  and  depth  of  testing.  One  example  of  this 
interdependent  combination  is  offered  by  tools  which 
provide  the  user  with  an  ability  to  create  test  cases 
from  requirements  "automatically”;  however,  to  collect 
information  regarding  the  number  of  inputs  successfully 
tested  and  inputs  with  successful  test  cases  will 
require  the  use  of  another  tool  normally  referred  to  as 
a  test  execution  tool.  This  interdependence  between 
tools  that  are  part  of  a  product  suite  can  make  data 
collection  for  metrics  more  difficult  and  costly. 
Ideally  the  tool  selected  will  be  designed  specifically 
for  coverage  and/or  complexity  analysis. 

g.  Fault  profiles.  Developer's  typically  have  a 
problem  reporting  system  in  place  whether  it  is  a 
manual  paper  system  or  an  automated  one.  These  systems 
provide  a  mechanism  for  an  organization  to  gather 
software  quality  measures  throughout  the  software  life 
cycle.  These  systems  can  usually  support  the  data 
requirements  of  the  fault  profiles  metric.  Depending 
upon  the  flexibility  of  the  system  in  place,  it  is 
possible  the  problem  reporting  system  can  be  adapted  to 
include  any  missing  data  collection  requirements  of  the 
fault  profiles  metric  that  are  not  already  in  place. 

If  an  adaptable  problem  reporting  system  is  not  already 
in  place  it  may  be  necessary  to  acquire  a  commercial 
system  for  collecting  and  reporting  software  problem 
reports.  Very  few  commercial  tracking  systems  exist 
because  many  large  organizations  develop  their  own 
software  trouble  reporting  systems  that  are  linked  to 
their  own  development  process.  The  commercial  systems 
have  to  be  developed  independent  of  any  methodology  in 
order  to  command  a  reasonable  market.  The  vendors  of 
the  fault  profiles  tools  listed  in  this  appendix 
maintain  that  their  product  is  process  independent  and 
that  they  collect  and  report  most  if  not  all  the  data 
elements  required. 

h.  Reliability.  A  number  of  software  reliability 
models  exist,  each  having  its  own  associated 
assumptions,  limitations,  and  applicability.  Many  of 
these  models  attempt  to  represent  the  uncertainty 
associated  with  software  performance  using  probability 
theory.  Statistical  methods,  such  as  confidence 
intervals,  are  used  to  make  inferences  about  the 
software's  performance.  Adopting  a  particular  model 
that  is  based  on  probability  theory  also  means 
accepting  the  particular  pattern  or  distribution  of 
software  failures  employed  by  that  particular  model. 

At  this  time  no  consensus  exists  regarding  the 
distribution  of  software  failures  within  a  system  and 
for  that  reason  this  appendix  does  not  endorse  the  use 
of  any  particular  model  to  supplement  the  collection 
and  reporting  of  the  reliability  metric.  The  data 
elements  of  the  reliability  metric  provide  some  of  the 
inputs  to  these  models.  The  data  elements  themselves 
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are  obtained  through  the  collection  of  observed 
software  and  system  failures  experienced  during 
testing.  The  candidate  reliability  tool  listed  in  this 
report,  the  Statistical  Modeling  and  Estimation  of 
Reliability  Function  for  Software  (SMERFS)  program,  can 
automatically  calculate  some  of  the  data  elements  used 
to  support  this  metric  as  well  as  providing  access  to 
many  reliability  models.  This  flexibility  encourages 
users  to  exercise  a  wide  variety  of  reliability  models, 
without  endorsing  a  particular  model,  in  order  to 
determine  which  provides  the  closest  representation  of 
the  actual  distribution  of  a  system's  software  failure 
rate. 

D-7.  Cost  tools 

The  following  products  support  collection  of  cost 
metric  data.  Those  data  elements  subject  to  automation 
are  marked  with* an  "X"  in  Table  D-l.  • 

a.  Artemis  I/CSCS 

b.  CA-SuperProject 

c.  Lotus  1-2-3  tProject  Resources 

d.  Micro  Planner  X-Pert  2.0 

e.  Multitrak 

f.  Primavera  5.0 

g.  Project  Scheduler  5 

h.  Tracker/3000 

i.  Ultra  Planner 

j .  Viewpoint 

D-8.  Schedule  tools 

The  following  products  support  collection  of  schedule 
metric  data.  Those  data  elements  subject  to  automation 
are  marked  with  an  "X"  in  Table  D-2. 

a.  Artemis  I/CSCS 

b.  CA-SuperProject 

c.  Lotus  1-2-3  §Project  Resources 

d.  Micro  Planner  X-Pert  2.0 

e.  Multitrak 

f.  Primavera  5.0 

g.  Project  Scheduler  5 

h.  Tracker/3000 

i.  Ultra  Planner 
•j.  Viewpoint 

D-9.  Computer  resource  utilisation  tools 
The  following  products  support  collection  of  computer 
resource  utilization  metric  data.  Those  data  elements 
subject  to  automation  are  marked  with  an  "X"  in  Table 
D-3. 

a.  DECps:  Data  Collector,  Performance  Advisor, 
Capacity  Planner,  and  Accounting  Chargeback 

b.  Laser  RX/UX  Performance  for  HP  9000  Systems 

c.  HP  GlancePlus  UX  Software  for  HP  9000  Systems 
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Tabla  D-l.  Caadldata  Cost  Matric  Tools 


Viewpoint 
Ultra  Planner 
Tracker/3000 
Project  Scheduler  5 
Pritnavera  5.0 


Multitrak 


Micro  Planner  X-Pert  2.0 


Lotus  1-2-3  8Project  Res. 


CA-SuperProject 


Artemis  I/CSCS 


BCWS  -  Total  #  of  dollars  that 
had  been  budgeted  for  the  work 
scheduled  to  be  accomplished  for 
the  identified  activity 


Bcwp  -  Total  /  of  dollars 
budgeted  for  the  wrk  actually 
performed  for  the  identified 
activity 


ACWP  -  Total  #  of  dollars 
actually  spent  for  the  work  done 
on  the  identified  activity 


D-10.  Softwara  anginaaring  anviroiu&aat  tools 
As  discussed  in  paragraph  D-6,  automated  tools  for 
collecting  or  reporting  SEE  metric  data  are  not 
feasible. 

D-ll.  Requirements  traceability  tools 
The  following  products  support  collection  of 
requirements  traceability  metric  data.  Those  data 
elements  subject  to  automation  are  marked  with  an  ”X" 
in  Table  D-4. 

a .  CHECKPOINT 

b.  RTM 

c .  RTrace 

D-12.  Requirements  stability  tools 
The  following  products  support  collection  of 
requirements  stability  metric  data.  Those  data 
elements  subject  to  automation  are  marked  with  an  "X" 
in  Table  D-5. 

a .  CHECKPOINT 

b.  RTM 

c.  RTrace 
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Tabla  D-2.  Candldata  Schadula  Matric  Tools 


Viewpoint 


Ultra  Planner 


Trac)ce]:73000 


Project  Scheduler  5 


Primavera  5.0 


Multitrak 


Micro  Planner  X-Pert  2.0 


Lotus  1-2-3  ^Project  Res. 


CA-SuperProject 


Artemis  I/CSCS 


Name  of  the  event,  activity  or 
milestone 


Date  on  which  event  is  planned  to 
start 


Date  on  which  event  is  planned  to 
complete 


Date  on  which  event  actually  began 


Date  on  which  event  actually 
completed  ■ 


Tabla  D-3.  Casdidata  Computar  Rasourca  Utilization 
Matric  Tools 


HP  GlancePlus  UX  Software  for  HP  9000  Systems 


Laser  RX/UX  Performance  for  HP  9000  Systems 


DECps:  Data  Collector,  Performance  ... 


For  each  CPU,  measured  value  of  CPU  capacity  utilized 
during  peak  operational  loading  periods  expressed  as  % 
of  the  CPU’s  total  capacity 


For  each  I/O  channel,  measured  value  of  I/O  channel 
capacity  utilized  during  peak  operational  loading 
periods  expressed  as  %  of  total  capacity  of  the  channel 


For  each  RAM,  measured  value  of  RAM  utilized  during 
peak  operational  loading  periods,  in  bytes 


For  each  storage  device,  measured  value  of  the  storage 
device  utilized  during  peak  operational  loading 
periods,  in  bytes 


Actual  amount  of  RAM  used  by  CSCX,  in  bytes 


Actual  amount  of  mass  storage  used  by  CSCX,  in  bytes 


’  Calculated  in  terms  of  disk  X/0  activity;  convert  by  calculating 
I/O  rate  for  each  drive  and  linking  to  corresponding  channel. 
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Tabla  D-4.  Candidata  Raquiranants  Traeaabllity  Jiatric  Tools 


RTrace 


CHECKPOINT 


#  of  Software  Peqt  Specificationg  (SRS)  per  CSCI 


#  of  SRS  not  traceable  to  Ueer  Functional  Description 
(UFD)  per  CSCI 


#  of  UFD  eoftware  reqts  traceable  to  SRS  per  CSCI 


#  of  SSS  (system  level  specification)  reqts  traceable 
to  SRS  per  CSCI  * 


#  of  CSCI’s  SRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI 


#  of  CSCI*s  SRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI's  CSCs 


#  of  CSCI’s  SRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI’s  CSUs 


#  of  CSCI’s  SRS  reqts  that  are  traceable  to  the 
computer  program  code  comprising  the  CSCI 


#  of  CSCI  *  s  recfts  from  the  SRS  covered  by  one  or  more 
documented  test  case 


#  of  CSCI’s  IRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI  • 


#  of  CSCI’s  IRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI’s  CSCs  * 


#  of  CSCI’s  IRS  reqts  that  are  traceable  to  the 
documented  design  of  the  CSCI’s  CSUs  * 


#  of  CSCI’s  IRS  reqts  that  are  traceable  to  the 
computer  program  code  comprising  the  CSCI  • 


#  of  CSCI’s  reqts  from  IRS(s)  covered  by  one  or  more 
documented  test  case  « 


#  of  Operational  Requirements  Document  (ORD)  reqts  for 
the  entire  system 


#  of  UFD  reqts  for  the  entire  system  • 


#  of  UFD  software  requirements  for  the  entire  system  I 


#  of  SSS  reqts  for  the  system  •] 


t  of  SSS  software  reqts  for  the  system  *1 


#  of  software  interface  reqts  for  the  system  * 


^  of  ORD  reqts  traceable  to  the  UFD 


#  of  UFD  reqts  traceable  to  the  ORD 


#  of  UFD  reqts  traceable  to  the  SSS  i 


w  of  UFD  reqts  traceable  to  ZRS(s)  * 


#  of  IRS  reqts  that  are  not  traceable  to  the  UFD  * 


#  of  SSS  software  reqts  traceable  to  IRS(s)  * 


•  Data  element  added  after  vendor  survey  data  received*  It  is 
unknown  whether  the  product  does  or  does  not  support  collection 
^  of  this  element. 

Completed  by  RTM  pointing  to  code  module. 
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Tabla  D-5.  Candidata  Raquiramanta  stability  Matric 
Tools 


RTrsce 

n 

RTM 

■ 

CHECKPOINT 

_ 

■ 

#  Software  Aeqt  Discrepancies  (SRD)  detected  to  date 

X 

D 

D 

#  of  SRDs  which  remain  open  to  date 

D 

D 

D 

#  of  SRDs  which  are  closed  to  date 

D 

D 

D 

#  of  ECP-S^  submitted  by  user  against  software  reqts 

D 

D 

D 

#  of  SLOC^  affected  by  approved  software  rsq[ts 
related  ECP-Ss  submitted  by  the  user 

X 

H 

■ 

#  of  CSUs  in  the  CSCI  which  are  affected  in  this 
reporting  period  by  approved  software  requirements-., 
related  ECP-Ss  submitted  by  the  user 

1 

■ 

1 

#  of  ECP-Ss  submitted  by  the  developer  egeinst 
software  rears 

■ 

H 

X 

#  of  SLOC  affected  by  approved  software  reqts 
related  ECP-Ss  submitted  by  the  developer 

H 

■ 

■ 

#  of  CSUs  in  the  CSCI  which  are  affected  in  this 
reporting  period  by  approved  software  recpairements- 
related  ECP-Ss  submitted  by  the  developer 

1 

1 

1 

#  of  SLOC  in  the  CSCI 

D 

El 

□ 

^  Engineering  Change  Proposal-Software 

^  Source  Line  of  Code  -  are  non-blank,  non-comment,  executable  and 
data  statements. 

^  Completed  by  RTM  object  containing  SLOC  attribute  per  code 
module. 


D-13.  Dasign  stability  tools 

The  following  products  support  collection  of  design 
stability  metric  data.  Those  data  elements  subject  to 
automation  are  mar)ced  with  an  "X”  in  Table  D-6. 

a.  CHECKPOINT 

b.  RTM 

c.  RTrace 

D-14.  Complexity  tools 

The  following  products  support  collection  of  complexity 
metric  data.  Those  data  elements  subject  to  automation 
are  marked  with  an  "X”  in  Table  D-7. 

a.  AdaMat 

b.  AdaQuest 

c.  Analysis  of  Complexity  (ACT) 

d.  CMS-2  Source  Code  Metrics  Generator 

e.  CodeMap 

f.  C-Metric 

g.  HINDSIGHT 
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Tabl«  D-6.  Caadidatft  Dasign  Stability  Matric  Tools 


RTrace 


RTM 


CHECKPOINT 


#  of  CSUb  that  are  planned  to  compriee  the  final 
delivery  of  the  CSCl 

i  of  CSUa  compriaing  the  CSCI  delivered  the  preaent 
aoftware  veraion 

t  of  CSUa  in  which  deaign  related  changea  %iere  made 
aince  the  laaf  delivery  of  the  CSCl 

#  of  CSUa  that  were  added  to  the  CSCl  with  aince  the 
laat  delivery 

0  of  CSUa  that  were  deleted  from  the  CSCl  since  the 
laat  delivery 


X 


X 


X 


X 


X 


X 


X 


X 


X 


h.  Logiscope 

i.  PC  Metric 

j.  QA  FORTRAN  and  QA  C 

k.  TESTBED 

l.  VIA/SmartDoc 

D-15*  Breadth  of  tasting  tools 

The  following  products  support  collection  of  breadth  of 
testing  metric  data.  Those  data  elements  subject  to 
automation  are  marked  with  an  "X"  in  Table  D-8. 

a.  CHECKPOINT 

b.  RTM 

c.  RTrace 

0-16.  Depth  of  tasting  tools 

The  following  products  support  collection  of  depth  of 
^ i^9 _ data.  Those  data  elements  subject  to 
automation  are  marked  with  an  "X”  in  Table  D-9. 

a.  AdaRAID 

b.  AdaTune 

c.  Analysis  of  Complexity  (ACT) 

d.  Autoflow  C 

.-e.  CMS-2  Test  Coverage  Analyzer 

f .  CodeMap 

g.  C-Metric 

h.  J73AVS  and  RXVP80 

i.  Logiscope 

j.  QA  FORTRAN  and  QA  C 

k.  TCAT  and  TCAT-PATH 

l .  TESTBED 


0-12 
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Table  D-7.  Candidate  Complexity  Metric  Tools 


^  Kill  include  these  metrics  in  updated  version  due  in  June  1992. 


D-17.  Fault  profiles  tools 

The  following  products  support  collection  of  fault 
profiles  metric  data.  Those  data  elements  subject  to 
automation  are  marked  with  an  "X”  in  Table  D-10. 

a.  CHECKPOINT 

b.  DDTs:  Distributed  Defect  Tracking 
C.  MIL/SOFTQUAL 

D-18.  Reliability  tools 

The  following  product  supports  collection  of 
reliability  metric  data.  Those  data  elements  subject 
to  automation  are  marked  with  an  "X”  in  Table  D-11. 
a.  SMERFS 
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Tabla  D-8.  Candidata  Braadth  of  Tasting  Matric  Tools 


RTrace 


RTH 


CHECKPOINT 


#  of  Software  Requirements  Specification  (SRS) 
documented  requirements  for  the  Computer  Software 
Configuration  Item  (CSC!)  under  development 

#  of  SRS  reqts  for  the  CSCI  that  have  been  tested 
using  approved  test  eases 

#  of  SRS  reqts  for  the  CSCI  that  have  been 
successfully  demonstrated  through  testing 

#  of  Interface  Requirements  Specification  (IRS) 

associated  with  the  CSCI  under  development 

#  of  IRS  reqts  for  the  CSCI  that  have  been  tested 
using  approved  test  cases 


#  IRS  reqts  for  the  CSCI  that  have  been 


successfully  demonstrated  through  testing  * 

#  of  Users'  Functional  Description  priority  level 

1  (UFDl)  reqts  associated  with  the  CSCI  under 
development  * 

#  of  UFDl  reqts  for  the  CSCI  that  have  been  tested 

using  approved  test  cases  • 

#  of  UFDl  reqts  for  the  CSCI  that  have  been 

successfully  demonstrated  through  testing  • 

#  of  UFD  priority  level  2-4  (UFD2-4)  reqts 

associated  with  the  CSCI  under  development 
(Separate  record  for  each  priority  level)  * 

#  of  UFD2-4  reqts  for  the  CSCI  that  have  been 

tested  using  approved  test  cases  • 

#  of  UFD2-4  reqts  for  the  CSCI  that  have  been 

successfully  demonstrated  through  testing  • 

Testing  phase  or  test  event  identifier  with  which 
the  preceding  data  elements  are  associated 


X 


X 


X 


*  Data  element  added  after  vendor  survey  data  received.  It  is 
un)cnown  whether  the  product  does  or  does  not  support  collection 
of  this  element. 


D-19.  Product  information 

Descriptions  of  the  tools  identified  in  the  previous 
paragraphs  are  presented  in  alphabetical  order  by 
product  name.  The  first  time  the  product  name  appears 
it  is  shown  in  bold  type  in  order  to  facilitate  lookup 
of  specific  products. 
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Tabla  D-9.  Candidata  Dapth  of  Taating  Matric  Tools 


1  For  calculation  requires  use  of  test  cases  and  test  verification 
together. 
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Tabla  D-lO.  Candidata  Fault  Profilas  Matrie  Tools 


MIL/SOFTQUAL 

DDTs 

■ 

1  -  CHECKPOINT 

1 

■i 

1  Total  priority  1  faulta  (all  racordt  per  CSCI) 

D 

li 

D' 

j Priority  1  fault!  due  to  requirement*  activity 

D 

a 

Priority  1  faults  due  to  design  activity  1 

D 

a 

D 

Priority  1  faults  due  to  coding  activity 

D 

D 

D 

Priority  1  faulta  due  to  documentation  activity 

D 

o 

D 

Priority  1  faults  due  to  other  activity 

a 

D 

Total  priority  1  faults  closed 

D 

a 

Total  priority  2-5  faulta  (Record  for  each 
priority) 

H 

X 

e 

Priority  2-5  faults  due  to  requirements  activity 

D 

a 

D 

Priority  2-5  faults  due  to  design,  coding, 
document  and  other  activities.  (Separate  record  for 
each  priority  class  w/in  each  activity) 

fl 

D 

D 

Total  priority  2-5  faults  closed  (Record  for  each 
priority  classification) 

1 

■ 

i 

Average  #  of  days  priority  1  fault*  remained  open 

D 

Average  #  of  days  priority  2-5  faults  remained 
open  (Record  for  each  priority  classification) 

H 

e 

Average  #  of  days  it  took  to  close  a  priority  1 
fault 

H 

■ 

Average  #  of  days  it  took  to  close  a  priority  2-5 
fault  (Record  for  each  priority  classification) 

H 

■ 

Average  age  of  priority  1  faults  (open  and  closed) 

D 

D 

Average  age  of  priority  2-5  faults  (open  and 
closed) 

■ 

H 

H 

a.  Tool.  AdaMXT 

(1)  Metric (s)  supported.  Complexity. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
DOS  Text  file. 

(4)  Platform(s) /language (s)  supported.  Ada  / 
DEC  VAX/ VMS,  SUN-3,  SUN-4,  Rational,  SCO  UNIX. 

(5)  Vendor  address/telephone. 

Dynamics  Research  Corporation 

60  Frontage  Road 
Andover,  MA  01810  USA 
1(800)  522-7321 
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Tabla  D-11.  Candidata  Raliability  Matric  Tools 


1  SMERFS 

1  Measured  software  failure  rate,  in  failures  per  month 

3 

Projected  software  failure  rate,  in  failures  per  month 

D 

Computed  point  estimate  of  mean  time  between  mission 
failures  caused  by  system  hardware  or  software  as  measured 
during  the  test  event 

1 

Calculated  80%  lower  confidence  bound  value  of  mean  time 
between  mission  failures  caused  by  system  hardware  or 
software  as  measured  during  the  test  event 

1 

Computed  point  estimate  of  mean  time  between  mission  fail¬ 
ures  caused  by  software  as  measured  during  the  test  event 

■ 

Calculated  80%  lower  confidence  bound  value  of  mean  time 
between  mission  failures  caused  by  software  as  measured 
during  the  test  event 

1 

Kean  time  to  restore  the  system  to  operational  condition  in 
hours 

■ 

Median  time  to  restore  the  system  to  operational  condition 
in  hours 

■ 

Maximum  95th  percentile  value  of  time  to  restore  the  system 
to  operational  condition 

■ 

(6)  Vendor  supplied  information.  AdaMAT  is  a 
comprehensive  static  source  code  analyzer  that  reports 
on  hundreds  of  Ada-specific  quality  metrics.  The 
metrics  focus  on  the  most  effective  use  of  the  Ada 
language  and  adherence  to  long-standing  software 
quality  engineering  principles.  AdaMAT  analyzes  Ada 
source  code  and  the  measurements  are  output  into 
detailed  reports  that  provide  visibility  into  the 
quality  of  the  code.  High-level  parameters  measure 
such  areas  as  reliability,  portability,  and 
maintainability.  While  other  metrics  address  specific 
programming  concerns,  such  as  code  simplicity, 
modularity,  self-descriptiveness,  exactness,  clarity, 
and  independence.  Output  can  also  be  displayed  in 
graphic  format  on  an  IBM  PC  or  compatible,  via  AdaMAT 's 
Metrics  Display  Tool, 
b.  Tool.  AdaQuest 

(1)  Metric (s)  supported.  Complexity. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  provided  in 
ASCII  file. 

(4)  Platform(s) /language(s)  supported.  Ada  / 

DEC  VAX/VMS,  ULTRIX  (Capability  due  in  4th  Quarter  FY 
1992) . 
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(5)  Vendor  address/telephone. 

General  Research  Corporation 

5383  Hollister  Ave 

Santa  Barbara,  CA  93460  USA 

(805)  964-7724 

(6)  .  Vendor  supplied  information.  AdaQuest  is  a 
computer-aided  software  engineering  (CASE)  tool  set 
supporting  Ada  software  tests  and  analysis  during  the 
coding,  testing  and  maintenance  phases  of  the  software 
life  cycle.  Through  a  combination  of  static  and 
dynamic  analysis,  AdaQuest  locates  potential  errors, 
measures  test  thoroughness,  and  assists  reverse¬ 
engineering  of  Ada  programs.  Static  Analysis  examines 
the  structural,  symbol  usage  and  control  flow 
characteristics  of  a  program,  and  produces  a  variety  of 
reports  from  this  information.  It  also  chec)cs  for 
violations  of  29  user-tailored  programming  standards. 
Dynamic  Analysis  instruments  the  source  code  of  a 
program  to  measure  execution  coverage  and  timing 
behavior,  and  also  checks  for  run-time  violations  of 
logical  assertions,  which  are  embedded  (as  Ada 
comments)  into  the  source  code  by  the  programmer.  A 
post-mortem  processor  then  analyzes  the  collected  data 
and  presents  it  as  tabular,  histogram  and  source 
listing  formats. 

c.  Tool.  AdaRAID 

(1)  Metric (s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation;  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  format  for 
electronic  media  unknown. 

(4)  Platform (s) /language (s)  supported.  DEC 
VAX/VMS  /  Ada,  JOVIAL. 

(5)  Vendor  address/telephone. 

PSS  Inc. 

429  Santa  Monica  Blvd. ,  Suite  430 
Santa  Monica,  CA  90401 
(310)  394-5233 

(6)  Vendor  supplied  information.  AdaRAID  is 
available  in  two  versions;  a  multi-processor  simulator 
version  and  a  hardware  version.  AdaRAID  can  debug 
programs  for  any  Ada  or  JOVIAL  compiler  that  provides 
debug  information  in  either  IEEE  695  or  PSS  Standard  or 
Blocked  format.  Users  can  view  source  and  monitor 
results,  registers  and  memory  via  conditional 
breakpoints  and  watchpoints;  as  well  as  with  step,  go 
and  stop  buttons;  time  programs;  trace  programs  all  in 
source  or  assembly  form. 

d.  Tool.  AdaTune 

(1)  Metric (s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation;  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports. 
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(4)  Platfonn(s) /language (s)  supported.  MS-DOS 
386  (32  BIT)  SUN-3 ,  Apollo  and  HP  9000/300  /  Ada. 

(5)  Vendor  address/telephone. 

Alsys 

67  South  Bedford  St. 

Burlington,  MA  01803-5152 
(617)270-0030 

(6)  Vendor  supplied  infornation.  AdaTune  is  a 
unique  software  engineering  tool  that  performs  both 
performance  analysis  and  coverage  analysis  of  Ada 
programs.  AdaTune  gives  you  greater  understanding  of 
your  applications  behavior.  AdaTune  gives  you 
quantitative  information  that  you  can  use  to  fine-tune 
your  application  performance  and  improve  its 
reliability. 

e.  Tool.  Analysis  of  Complexity  (ACT) 

(1)  Metric (s)  supported.  Complexity,  depth  of 

testing.  •  . 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file. 

(4)  Platform (s) /language (s)  supported. 

SUNSparc,  SUN,  IBM  RS/6000,  DEC  VMS,  DEC  RISC,  Apollo, 
HP,  Silicon  Graphics,  IBM  PC  and  Compatibles  /  C, 
COBOL,  PDL  81,  OS  PL/1,  VAX  FORTRAN,  PASCAL,  Ada,  CMS- 
2,  BLISS  (DEC),  C++. 

(5)  Vendor  address/telephone. 

McCabe  &  Associates,  Inc. 

5501  Twin  Knolls  Road,  Suite  111 
Columbia,  MD  21045  USA 
(301)  596-3080 

(6)  Vendor  supplied  information.  ACT  provides 
visual  representations  of  code  that  ma)ce  identifying 
complex  code  easy.  ACT  generates  a  complete  set  of 
test  paths  and  end-to-end  test  conditions  that  ensure 
100%  basis  path  coverage.  ACT  increases  productivity 
by  avoiding  redundant  tests  and  reducing  the  costly 
time  it  takes  to  create  test  paths  by  hand.  With  ACT, 
unit  testing  can  take  l/lOO  of  the  time  it  takes  using 
ad  hoc  or  conventional  methods.  ACT  computes  the 
McCabe  Cyclomatic  Complexity  Metric  for  each  module; 
application  of  this  metric  saves  a  project  as  much  as 
50%  in  testing  costs. 

f.  Tool.  Artemis  X/C8C8 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

(4)  Platform (s) /language (s)  supported. 

(a)  PC  and  Vax  versions  and  the  newly 

announced  Unix  version,  I/CSCS  is  also  available  for 
mainframe  computers. 
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(b)  Artemis  I/CSCS  Mini  2.6  is  now  available 
on  Unix  platforms,  including  the  Hewlett-Packard  HP9000 
and  Sun  SPARCstation. 

(5)  Vendor  address/ telephone. 

Lucas  Management  Systems 

12701  Fait  Lakes  Circle 
Suite  350 

Fairfax,  Va,  22033 
(703)  222-1111 
FAX  (703)  222-8203 

(6)  Vendor  supplied  information.  I/CSCS 
(Integrated  Cost  Schedule  Control  System)  is  meant  to 
help  control ■ costs  in  any  business  or  project.  It 
helps  managers  estimate,  organize,  plan,  and  analyze 
project  performance  from  proposal  through  delivery. 

The  software  complies  with  the  DODI  5000.2  management 
reporting  system  standard  specified  in  some  federal 
contracts.-  The  newest  personal  computer  vef'sion  of  the 
package  works  with  Microsoft  Windows  3.0,  taking 
advantage  of  Windows'  graphical  user  interface, 
extended  memory,  cut-and-paste  among  applications,  and 
the  ability  to  query  databases  from  within  an 
application  using  the  SQL  (structured  query  language) 
standard.  I/CSCS  PC  2.0  also  includes  Artemis 
Presents!,  a  Postscript  graphics  tool  designed  to  help 
users  combine  charts  and  reports.  It  works  with 
Ethernet  networks  and  distributed  databases,  and  the 
newest  version  offers  a  choice  of  support  for  the 
Oracle  or  Ingres  database  software. 

g.  Tool.  Autoflov  c 

(1)  Metric (s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file.  Postscript,  HPGL  Plotter,  lEW/ADW  import 
file,  Lotus  PIC  file. 

(4)  Platform(s) /language(s)  supported.  MS-DOS, 
OS/2,  UNIX:  C. 

(5)  Vendor  address/telephone. 

AutoCase  Technology 

10133  S  Portal  Ave. 

Cupertino,  CA  95014 
(408)  446-2273 

(6)  Vendor  supplied  information.  Generates  flow 
charts  and  structure  charts  automatically  from  existing 
C  code.  It  also  performs  automatic  test  coverage 
analysis.  It  takes  the  hassle  out  of  trying  to 
decipher  someone  else's  logic  in  a  program.  It  also 
saves  time  when  it  comes  to  making  your  project 
documentation  complete.  No  limit  on  source  program 
size. 
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h.  Tool.  CA-SuparProjact 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

(4)  _  Platform (s) /language (s)  supported. 

(a)  Windows,  MS-DOS,  IBM  mainframes,  Unix, 
Apple  Macintoshes  and  VAX/VMS. 

(5)  Vendor  address /telephone. 

Computer  Associates 

12120  Sunset  Hills 
Reston,  VA  22091 
(800)  874-9392 

(6)  Vendor  supplied  information. 

CA-Super Project  is  a  project  management  program  that  is 
rich  in  functionality  and  highly  professional.  The 
product,  first  introduced  in  1984,  currently  holds 
two-fifths  of  the  mar)cet  for  project  management 
software.  CA-SuperProject  handles  resource  management 
effectively,  even  for  multiple  projects.  The  default 
layout  is  a  Task  Outline,  which  has  a  spreadsheet 
interface  and  includes  a  Gantt  chart.  The  alternatives 
are  a  PERT  chart  and  a  Work  Breakdown  Structure  chart. 

A  cost/resource  histogram  can  also  be  displayed  in  a 
panel.  With  each  type  of  chart,  you  can  define  the 
task  dependencies  and  durations,  either  with  the  mouse 
or  via  a  forms  interface.  The  Accounts  Outline  focuses 
on  the  costs  of  the  project  and  lets  you  group  the 
various  tasks  by  account  code.  If  you  set  up  your 
codes  to  reflect  different  types  of  resource,  the 
accounts  outline  shows  how  much  you're  spending,  for 
instance,  on  wages  to  contract  staff,  part-timers,  or 
full-time  employees,  and  how  much  on  equipment  hire. 
Data  can  be  imported  and  exported  in  a  variety  of 
formats,  including  Lotus  1-2-3,  dBase  III,  Microsoft 
Excel,  SuperCalc,  SYLK,  Fixed  ASCII  and  Comma  Separated 
Values  (CSV)  as  well  as  two  other  project  management 
programs,  Abtex's  PertMaster  Advance  and  Hoskyn's 
Project  Manager  Workbench. 

i .  Tool .  CHECKPOINT 

(1)  Metric (s)  supported.  Requirements 
traceability,  requirements  stability,  design  stability, 
breadth  of  testing,  fault  profiles. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file  suitable  for  storage  in  personal  repository. 

(4)  Platform (s) /language (s)  supported. 
Application  supports  all  platforms  and  over  300 
languaqe'5.  It  operates  under  MS  DOS  and  will  be 
available  in  UNIX  in  the  near  future. 
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(5)  Vendor  address/telephone. 

SPR  Inc. 

77  South  Bedford  St. 

Burlington,  MA  01803-5154 
(617)  273-0140 

(6)  _  Vendor  supplied  information.  CHECKPOINT  is 
a  powerful  knowledge-based  tool  that  provides  guidance 
and  support  for  software  managers  and  information 
systems  executives  seeking  to  improve  the  quality  of 
their  products,  shorten  delivery  times  and  lower  both 
development  and  maintenance  costs  by  increasing 
productivity.  CHECKPOINT  automates  and  integrates 
support  for  three  critical  management  skills  in  one 
package:  comprehensive  measurement,  dependable 
estimation  for  project  planning,  and  expert  assessment 
for  tradeoff  strategies. 

j.  Tool.  CM8-2  8ouree  Code  Metrics  Generator 

(1)  Metric (s)  supported.  Complexity 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  provided  in 
text  files. 

(4)  Platform (s) /language (s)  supported.  CMS-2  / 
VAX /VMS,  MS-DOS. 

(5)  Government  address /telephone. 

Fleet  Combat  Direction  Systems  Support  Activity 
San  Diego,  CA 
(703)  671-1475 

(6)  Government  supplied  information.  The  tool 
operates  independently  of  any  programming  standards 
and/or  conventions.  In  processing  raw  CMS-2  the  tool 
develops  metric  and  statistic  reports  including 

call tree  hierarchy  report,  keyword  frequency  report, 
source  code  statistics  report,  list  data  units  by  type 
and  alphabetize  data  units. 

k.  Tool.  CM8-2  Test  Coverage  Analyzer 

(1)  Metric (s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  provided  in 
text  file. 

(4)  Platform (s) /language (s)  supported. 

(a)  VAX/VMS,  MS-DOS  /  CMS-2. 

(5)  Government  address/telephone. 

Fleet  Combat  Direction  Systems  Support  Activity 
San  Diego,  CA 
(703)  671-1475 

(6)  Government  supplied  information.  The  tool 
performs  flow  analysis  of  the  source  code,  inserting 
instrumentation  code  at  the  respective  branch  points. 
Execution  of  the  instrumented  code  during  unit  and/or 
integration  test  causes  development  of  a  database  from 
which  reports  are  generated  identifying  the  percentage 
of  branch  points  exercised,  the  specific  exercised 


D-22 

in- 


30  September  1992 


DA  Pam  73-1,  Part  Seven  (Draft) 


branch  points,  and  repetition  counts  for  specific 
branch  points. 

l .  Tool .  CodeMap 

(1)  Metric (s)  supported.  Complexity,  depth  of 
testing. 

(2)  -  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  provided  in 
ASCII  file  suitable  for  storage  in  personal  repository. 

(4)  Platform (s) /language (s)  supported. 

Platform (s)  unknown  /  Ada,  C. 

(5)  Vendor  address/telephone. 

Cadre  Technologies 

2111  Wilson  Blvd. 

Suite  700 

Arlington,  VA  22201 
(703)  875-8670 

(6)  Vendor  supplied  information.  (No 
information  provided) . 

m.  Tool.  C-Metric 

(1)  Metric (s)  supported.  Complexity,  depth  of 
testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  format  for 
electronic  media  unknown. 

(4)  Platform(s) /lan^age(s)  supported.  DOS, 

OS/2  /  C,  C++. 

(5)  Vendor  address/telephone. 

Software  Blacksmiths 

6064  St.  Ives  Way 
Mississauga,  Ontario 
Canada,  L5N  4M1 
(416)  858-4466 

(6)  Vendor  supplied  information.  C-Metric 
calculates  path  complexity  and  counts  code,  statements, 
and  comments. 

n.  Tool.  DDTs:  Distributed  Defect  Tracking 

(1)  Metric (s)  supported.  Fault  profiles. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s) . 

(3)  Output.  Text  reports.  Output  flat  file 
format  for  electronic  media. 

(4)  Platform(s) /language(s)  supported.  SUN-3, 
SUN-4,  HP-9000,  RS-6000,  SCO  UNIX,  DECstation  or 
VAXstation  running  Ultrix,  Apollo  running  the  vendor 
supplied  UNIX. 

(5)  Vendor  address/telephone. 

QualTrak 

1250  Oakmead  Parkway,  Suite  210 
Sunnyvale,  CA  94088-35994 
(617)  273-0140 

(6)  Vendor  supplied  information.  DDTs  keeps 
defects  organized  and  separated  into  projects.  Each 
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defect  record  keeps  all  available  information  regarding 
the  bug  in  one  neat  package,  including  the  current 
status  of  the  defect,  the  data  files  needed  to 
reproduce  the  problem,  text  files  of  any  length  to 
explain  the  problem,  or  workarounds  from  the  lab 
engineer.  .  The  distributed  operation  of  DDTs  is  vital 
if  code  developers  are  in  different  locations  or  are 
implementing  on  different  machines.  DDTs  supports 
software  development  conforming  to  DOD-STD-2167A  and 
the  IEEE  P1044  draft  proposed  defect  tracking  system 
standard.  ^ 

o.  Tool,  pzcps:  Data  Collector#  Performance 
Advisor#  Capacity  Planner#  and  Accounting  Chargeback 

(1)  Metric (s)  supported.  Computer  resource 
utilization. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table(s). 

(3)  Output.  Textual  and  (Graphic  output. 

Provides  output  in  ASCII  or  DOS  text  file  suitable  for 
storage  in  personal  repository. 

(4)  Platform(s) /language (s)  supported.  DEC 
VAX /VMS. 

(5)  Vendor  address /telephone. 

Digital  Electronics  Corp. 

PO  Box  2008 

Nashua  NH  03061-2008 

1(800)  344-4825 

(6)  Vendor  supplied  information.  DECperformance 
Solution  (DECps)  is  an  integrated  set  of  VMS  layered 
products  that  provide  performance  and  capacity 
management.  The  tools  provide  "what  if"  analysis 
allows  the  user  to  preview  the  effects  of  changes  in 
operation  before  they  are  made.  Resource  accounting  is 
easily  accomplished  by  generating  a  report  of  changes 
based  on  system  utilization.  The  DECps  Performance 
Advisor  provides  performance  analysis.  The  DECps 
Capacity  Planner  software  determines  system  performance 
for  various  workloads  and  configurations.  The  Data 
Accounting  Chargeback  software  allocates  charges  for 
resource  usage.  The  Data  Collector  software  ties  the 
other  three  elements  of  the  product  suite  together  by 
gathering  and  managing  VMS  system  data  according  to 
user  specified  requirements.  Data  collection  takes 
place  at  each  node  according  to  a  user  defined 
schedule. 

p.  Tool.  HINDSIGHT 

(1)  Metric(s)  supported.  Complexity. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 

ASCII  file.  ^ 

(4)  Platform (s) /language (s)  supported.  SUN 
4.0.3+,IBM  RS/6000,  HP  9000,  Apollo,  DEC  /  C,  FORTRAN 
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(available  in  3rd  QTR  of  1992),  (available  in  4th 

QTR  of  1992). 

(5)  Vendor  address/telephone. 

Advanced  Software  Autonation  Inc. 

2880  Lakeside  Drive  Suite  226 
Santa  Claiza,  Ca,  95054 

(408)  492-1668 

(6)  Vendor  supplied  information.  HINDSIGHT  is  a 
complete,  fully  integrated  software  maintenance 
environment.  It  automates  software  maintenance  and 
testing,  structure  charts,  logic  flow  diagrams,  test 
coverage  analysis,  performance  analysis,  automatically 
generated  reports,  and  global  VAL  analysis. 

g.  Tool.  EP  Gla&eePlus  UX  Software  for  EP  9000 
Systems 

(1)  Metric (s)  supported.  Computer  resource 
utilization. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Provides  graphical  displays  in 
real-time. 

(4)  Platform(s) /language (s)  supported.  HP  9000 
Series  including  Series  300,  400,  600,  700,  800, and  LAN 
conf igurations . 

(5)  Vendor  address/telephone. 

Hewlett-Packard 

Building  51L 

5301  Stevens  Creek  Blvd. 

Santa  Clara,  CA  95052-8059 
1(800)  637-7740 

(6)  Vendor  supplied  information.  HP  GlancePlus 
is  an  easy-to-use  tool  for  viewing  HP-UX  system 
performance.  It  enables  system  administrators  to 
examine  system  activity,  identify  and  resolve 
occasional  performance  bottlenecks  when  they  take 
place.  Bar  graphs  show  overviews  of  data,  including 
CPU,  disk,  memory,  and  swap  utilization.  Also  provides 
Network  performance  information  including  Network  File 
System  and  LANs. 

r.  Tool.  J73AVS  and  RXVP80 

(1)  Metric (s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s) . 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file. 

(4)  Platform(s) /language(s)  supported. 

(a)  RXVP80;  VAX  /  FORTRAN. 

(b)  J73AVS:  VAX  IBM  MAINFRAME  /  Jovial. 

(5)  Vendor  address/telephone. 

General  Research  Corporation 

5383  Hollister  Ave 

Santa  Barbara,  CA  93460  USA 

(6)  Vendor  supplied  information.  J73AVS  is  a 
JOVIAL  J73  verification  and  validation  tool  and  RXVP80 
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is  a  FORTRAN  verification  and  validation  tool.  These 
tools  verify  semantic  consistency  of  programs.  They 
produce  reports  based  on  static  and  dynamic  analysis, 
including  cross  reference,  calling  tree,  and  timing. 

GRC  offers  training,  support,  and  customization. 

s.  Tool.  Laser  RZ/UZ  Psrfermaaes  Maaagsas&t 
Software  for  EP  9000  Systems 

(1)  Metric(s)  supported.  Computer  resource 
utilization. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

p)  Output.  Provides  output  in  ASCII  or  DOS 
text  file  suitable  for  storage  in  personal  repository. 

(4)  Platform(s) /language(8)  supported.  HP  9000 
Series  including  Series  300,  400,  600,  700,  800,  and 
LAN  configurations. 

(5)  Vendor  address/telephone. 

Hewlett-Packard 

Building  51L 

5301  Stevens  Creek  Blvd. 

Santa  Clara,  CA  95052-8059 
1(800)  637-7740 

(6)  Vendor  supplied  information.  Laser  RX/UX 
allows  you  to  quickly  isolate  and  analyze  system  CPU, 
memory,  and  I/O  bottlenecks,  efficiently  monitor  and 
analyze  computing  resource  usage  over  time,  easily 
document  system  growth  and  plan  for  future  needs,  and 
uniformly  establish  and  monitor  service  levels. 

t.  Tool.  Logiseope 

(1)  Metric (s)  supported.  Complexity,  depth  of 
testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file. 

(4)  Platform (s) /language (s)  supported.  HP  9000, 
SUN-3,  SUN-4,  Sparc,  Apollo,  VAX  VMS,  IBM  RISC,  DEC 
ULTRIX  and  others  contact  Logiseope  for  details  /  Over 
80  Languages  and  Dialects  supported.  Contact  Logiseope 
for  details. 

(5)  Vendor  address/telephone. 

Verilog  Inc. 

3010  LBJ  Freeway,  Ste.  900 
Dallas,  TX  75234 
1(800)  424-3095 

(6)  Vendor  supplied  information.  Logiseope  is  a 
source  code  analyzer  that  provides  complexity  analysis 
and  test  coverage  analysis.  Logiseope  supports 
development,  testing,  maintenance,  and  reverse 
engineering  activities.  Logiseope  can  generate  DOD- 
STD-2167A  standard  documents,  Kiviat  Diagrams,  Criteria 
Graphs,  and  Source  Code  Control  Graph.  Logiseope 
calculates  and  reports  over  thirty  different  languages 
including  Ada,  C,  COBOL,  FORTRAN,  Pascal  and  PL/1. 


D-26 


30  B«pt«mb«r  1992 


DA  Pun  73-1,  Part  Sovan  (Draft) 


u.  Tool.  Lotus  1-2-3  9Projact  Rasoxireas 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

( 4 )  .  Platform ( s ) / language ( s )  supported . 

(5)  Vendor  address/ telephone. 

Convergics  Corp. 

7910  Ivanhoe  Ave.  Suite  405 
La  Jolla,  CA  92037 
(619)  689-2433 

(6)  Vendor  supplied  information. 

§Project /Resources  add-in  for  1-2-3  Release  2.x  lets 
you  access  standard  project-management  functions  via 
Lotus-style  menus.  You  begin  by  entering  information 
on  the  resources,  tas)cs,  and  constraints  associated 
with  your  project,  using  the  add-in’s  45  §f unctions 
when  necessary.  fDURA,  for  example,  returns  the 
duration  of  a  tasJc,  and  §FREEFLOAT  returns  its  slac)c 
time.  With  €Project/Resources,  you  can  do 
critical-path  analysis,  determine  earliest  and  latest 
stsi^t/f inish  dates,  and  produce  schedules  by 
resource-limited,  resource-level,  and  tas)c-precedence 
methods.  The  pac}cage  includes  a  wor)c-time  calendar,  as 
well.  §Project/Resources  produces  standard 
project-management  graphs,  including  Gantt  charts.  Pert 
diagrams,  and  histograms.  Graphs  can  be  saved  to  dis)c 
as  PIC  files  and  printed  by  using  1-2-3 's  PrintGraph 
program. 

V.  Tool.  Micro  Planner  Z-Pert  2.0 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

(4)  Platform (s) /language (s)  supported. 

(5)  Vendor  address/telephone. 

Micro  Planning  International  Inc. 

3801  East  Florida  Ave.  Suite  605 
Denver,  CO  80210 

(415)  389-1420 

(6)  Vendor  supplied  information.  Micro  Planner 
X-Pert  2.0  supports  up  to  10,000  activities,  50 
subprojects  and  unlimited  interfaces  between  projects. 
The  pac)cage  manages  large  and  complex  projects  with 
easy  viewing,  tracking  and  control.  X-Pert  2.0  handles 
multi-project  cost,  resource  and  schedule  analysis. 
Users  can  customize  reports  and  screens  by  annotating 
text  and  graphics.  They  open  up  into  X-Pert 's  'inner 
desktop, '  where  folders  and  icons  representing  projects 
can  be  organized.  Starting  with  an  existing  report 
layout,  users  can  modify  and  create  their  own  report 
format,  then  save  it  as  an  icon.  Version  2.0  includes 
MacPlot  Professional,  a  third-party  plotter  driver  that 
supports  most  plotters  such  as  Hewlett-Packard,  Houston 
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Instruments  and  CalComp.  X-Pert  1.04  users  who  have 
the  MPI  annual  support  program  can  upgrade  to  2.0. 
w.  Tool.  MIL/SOFTQUAL 

(1)  Metric (s)  supported.  Fault  profiles. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Provides  output  in  ASCII  or  DCS 
text  file  suitable  for  storage  in  personal  repository. 

(4)  Hardware /Software  Required.  Operates  on  all 
compatible  IBM  PCs,  XTs  or  ATs  with  a  minimum  of  64 OK 
RAM  and  a  hard  dis)c. 

(5)  Vendor  address /telephone. 

The  Carmen  Group  Inc. 

PO  Box  867689 

Plano,  TX  75086-7689 

(214)867-5089 

(6)  Vendor  supplied  information.  MIL/SOFTQUAL 
is  an  unique  software  development  quality  cost  tool 
which  helps  one  comply  with  the  cost  and  quality 
requirements  of  MIL-Q-9858A,  and  with  DOD-STD-2167A. 
MIL/SOFTQUAL  was  designed  to  line  up  exactly  with  the 
DOD-STD-2167A  development  activities. 

X.  Tool.  Multitrak 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

(4)  Platform (s) /language (s)  supported. 

(a)  IBM  mainframe 

(5)  Vendor  address/telephone. 

Multitrak  Software  Development  Corp. 

119  Beach  St. 

Boston,  MA  02111 
(617)  482-6677 

(6)  Vendor  supplied  information.  Multitrak 
provides  "work  management"  facilities  that  function 
from  a  single  centralized  database.  This  database 
houses  all  project  and  resource  information  to 
facilitate  data  sharing  among  some  200  users. 

Multitrak  describes  work  management  as  an  extension  to 
conventional  project  management  techniques.  Wor)c 
management  provides  an  infrastructure  for 
enterprise-wide  planning,  budgeting,  management  and 
evaluation  of  all  projects  and  people  in  the  IS 
organization.  Data  on  the  mainframe  is  available  in 
real  time;  no  uploading  or.  downloading  is  required 
between  a  workstation  and  a  host,  eliminating 
synchronization  problems  and  enhancing  data  integrity. 
Through  Multitrak 's  multiproject  architecture,  project 
and  resource  information  can  be  collected  at  any  level 
of  the  specified  work  breakdown  structure,  simplifying 
the  task  of  aggregating  and  summarizing  information  for 
management  reports. 


D-28 


n  -  ms 


30  6«ptttmb«r  1992 


DA  Pan  73-1,  Part  Savan  (Draft) 


y.  Tool.  PC  Matric 

(1)  Metric (s)  supported.  Complexity. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Utilities  are 
provided  to  create  comma  delimited  files  from  the 
reports.  These  files  can  be  imported  into  database  and 
spreadsheet  products. 

(4)  Platform (s) /language (s)  supported. 

(a)  DOS  /  Ada,  Assembler,  dBase,  C, 

COBOL,  FORTRAN,  Modula-2,  Pascal,  QuickBASIC. 

(b)  UNIX  /  Ada,  C,  C++,  UNIX/386,  SUN  3/50, 
SUN  386i,  SUNSparc. 

(C)  VAX  /  FORTRAN. 

(d)  MACINTOSH  /  Pascal. 

(5)  Vendor  address/telephone. 

SET  Laboratories,  Inc. 

PO  Box  868 
Mulino,  OR  97042 
(503)  829-7123 

(6)  Vendor  supplied  information.  PC-Metric  and 
UX  Metric  is  a  software  measurement  tool  used  to 
analyze  source  code.  It  produces  two  reports,  1)  the 
complexity  analysis  report  which  lists  metrics  values 
calculated  for  each  function  and  the  combined  values 
for  the  entire  module  under  consideration,  and  2)  the 
exceptions  report  which  highlights  all  measured  values 
that  lie  outside  of  predetermined,  user-defined  limits. 
This  tool  is  intended  for  two  types  of  users,  the 
software  developer  and  the  software  project  manager. 

The  manual  is  divided  into  three  parts:  Part  1  is  a 
tutorial  on  the  field  of  software  metrics  including  an 
annotated  bibliography;  Part  2  describes  the 
installation,  configuration  and  use  of  the  product;  and 
Part  3  instructs  users ''on  what  to  do  with  the  data. 

z.  Tool.  Primavera  Project  Planner  Version  5.0 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Graphics  and  text  based  reports. 

(4)  Platform (s) /language (s)  supported. 

(a)  640K  RAM,  DOS  3.1  or  later.  386-based  PC 

or  better  and  2MB  disk  space  recommended. 

(5)  Vendor  address/telephone. 

Primavera  Systems  Inc. 

Two  Bala  Plaza 
Bala  Cynwyd,  Pa,  19004 
(800)  423-0245 
(215)  677-8600 

(6)  Vendor  supplied  information.  Primavera 
Project  Planner,  Version  5.0  offers  multiple  project 
options,  scheduling  of  up  to  100,000  activities  per 
project,  and  a  maximum  project  span  of  80  years.  It 
includes  resource  management  functions  that  allow  users 
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to  define  up  to  16  different  application  profiles  and 
specify  which  profile  patterns  should  be  used  when 
allocating  various  resources  to  project  activities. 
Users  can  define  custom  data  items  to  track  cost, 
project  revenue,  planned  starting  dates,  and  levels  of 
difficulty.  An  office  can  assemble  schedules  of  the 
activities,  which  range  from  design  reviews  and 
software  testing,  to  demonstration,  conversion, 
debugging  and  implementation.  The  schedules  show  major 
reviews,  commitment  dates,  key  project  constraints  and 
project  dependencies.  Additional  new  features  include 
a  tables  format  that  allows  entry  of  time  sheet 
information  for  resources  and  a  custom  report  writer. 
The  package  supports  several  networks  including  NetWare 
and  VINES  and  can  import  and  export  Lotus  1-2-3,  dBASE, 
and  Excel  files. 

aa.  Tool.  Project  Scheduler  5 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table(s). 

(3)  Output. 

(4)  Platform (s) /language (s)  supported. 

(5)  Vendor  address/telephone. 

Scitor  Corp. 

250  Lincoln  Centre  Dr. 

Foster  City,  CA  94404 
(415)  570-7700 

(6)  Vendor  supplied  information.  Project 
Scheduler  5  provides  the  user  with  the  capability  to 
manage  multiple  projects.  You  can  link  projects 
together  in  a  way  that  establishes  cross-project 
dependencies  by  task  but  maintains  the  independence  of 
the  projects  involved.  This  means  that  a  manager  can 
work  with  a  project  and  try  to  resolve  schedule  and 
resource  problems  independently  before  across-project 
schedule  is  updated.  Individual  managers  are  flagged 
that  agreements  they've  made  to  other  managers  have 
slipped,  but  only  when  the  linked  projects  are  all 
brought  into  memory  is  the  cross-project  schedule 
updated.  Projects  can  be  dependent  on  each  other  even 
if  they  don't  share  resources. 

bb.  Tool.  QA  FORTRAN  and  QA  C 

(1)  Metric(s)  supported.  Complexity,  depth  of 
testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file. 

(4)  Platform (s) /language (s)  supported. 

(a)  VMS,  ULTRIX,  SUN  3,  SUN  4,  SUNSparc 
(OS3.5  or  later),  CRAY  UniCOS,  HP  9000-300/800,  Convex 
COS,  Alliant  FX40-81,  FX2800,  Solbourne,  Silicon 
Graphics  Personal  Iris  workstations  /  FORTRAN. 
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(b)  ULTRIX,  SUN  3,  SUN  4 ,  SUNSparc  (OS3.5  or 
later),  CRAY  UniCOS,  HP  9000-300/800,  Convex  COS, 
Alliant  FX40-81,  FX2800,  Solbourne,  Silicon  Graphics 
Personal  Iris  wor)tstations  /  C. 

(5)  Vendor  address /telephone. 

PROGRAMMIH.G  RESEARCH  CORPORATION 
Suite  520 

8701  Bedford-Euless  Road 
Hurst,  TX  76053 
(817)589-0949 

(6) .  Vendor  supplied  information.  QA  C  is  a 
toolset  for  analyzing,  improving,  and  maintaining  C 
programs.  Based  on  ANSI  validated  technology  to  chec)c 
for  over  500  reliability,  portability,  and  maintenance 
issues  within  the  C  Language.  QA  FORTRAN  is  a  toolset 
for  analyzing,  improving,  and  maintaining  FORTRAN 
programs.  Checks  for  hundreds  of  reliability, 
portability,  and  maintenance  issues  and  computes  28 
quality  metrics  about  the  code. 

cc.  Tool.  RTM 

(1)  Metric (s)  supported.  Requirements 
traceability,  requirements  stability,  design  stability, 
breadth  of  testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file  suitable  for  storage  in  personal  repository. 

(4)  Platform(s) /language(s)  supported. 
Application  supports  all  platforms.  It  operates  under 
SUN,  DEC  or  HP  systems. 

(5)  Vendor  address/telephone. 

Marconi  Systems  Technology 

1111  Jefferson  Davis  Hwy 
Arlington,  VA  22202 
(703)  920-7581 

(6)  Vendor  supplied  information.  RTM  supports 
entire  SDLC.  Data  base  is  a  VCRI  (Verification  Cross 
Reference  Index) .  Requirements  stripping  includes 
transferring  requirements  into  the  data  base.  Users 
may  scan  original  requirements  documents  by  line  or 
string  search.  Engineering  requests  are  classified 
into  keywords.  Interface  with  Teamwork,  Software 
through  Pictures  CASE  tools  and  others.  Supports  DOD- 
STD-2167A  format. 

dd.  Tool.  RTrace 

(1)  Metric (s)  supported.  Requirements 
traceability,  requirements  stability,  design  stability, 
breadth  of  testing. 

(2)  Data  elements  subject  to  automation:  Sea 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file  suitable  for  storage  in  personal  repository. 

(4)  Platform(s) /language(s)  supported.  SUN  3, 
SUN  4,  VAX/VMS,  IBM  RS6000.  Interfaces  with  teamwork 
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and  Software  Through  Pictures. 

(5)  Vendor  address /telephone. 

PROTOCOL 

500  International  Drive 
Mt.  Olive,  NJ  07828 
(201)  347-7900 

(6)  Vendor  supplied  information.  RTrace  is  a 
powerful  requirements  management  tool.  It  supports  all 
phases  of  a  system's  life  including  design, 
development,  test  and  post  deployment  change 
evaluation. 

ee.  Tool.  Statistical  Modeling  and  Estimation  of 
Reliability  Function  for  Software  (SMERF8) 

(1)  Metric (s)  supported.  Reliability., 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports. 

(4)  Platform (s) /language (s)  supported.' 
Application  supports  all  platforms  and  runs  under  any 
operating  system  on  any  type  of  computer  with  a  Fortran 
compiler. 

(5)  Vendor  address /telephone. 

Naval  Surface  Warfare  Center 
Dahlgren,  VA  22448 

(703)  663-7927 

(6)  Government  supplied  information.  SMERFS  is 
a  series  of  reliability  modeling  tools  developed  to 
accommodate  a  variety  of  environmental  and  behavioral 
characteristics  experienced  by  software  intensive 
systems  during  testing.  The  SMERFS  performs  five 
different  )cinds  of  operations  on  software  failure  data: 
data  transformations,  data  statistics,  plots  of  the 
original  data,  processing  of  data  through  one  of  ten 
reliability  models,  and  an  analysis  of  the  "goodness  of 
fit"  between  the  actual  testing  data  and  the  failure 
rate  projections  of  each  model  used.  The  selection  of 
a  reliability  model  within  SMERFS  depends  upon  two 
factors:  the  timing  mode  used  for  tracking  error  data 
(these  include  clock  time,  computer  time  or  testing 
interval  time)  and  the  assumptions  surrounding  the  use 
of  a  particular  model.  Once  the  selection  of  a  model 
is  limited  to  one  of  the  two  groups  based  on  the  timing 
mode  used,  the  user  must  then  consider  the  assumptions 
underlying  the  use  of  each  model  within  that  group 
before  making  a  final  selection.  (These  assumptions 
are  described  in  the  SMERFS  User's  Manual  NAVSWC  TR  84- 
373).  The  use  of  clock  time  and/or  computer  time  for 
software  failure  measurement  presupposes  the  use  of  one 
of  the  five  following  models:  Littlewood  and  Verral 
Bayesian,  Musa  Basic  Execution  Time,  Geometric, 
Nonhomogenous  Poisson  Process  (for  time  between  error 
occurrence),  or  the  Musa  Log  Poisson  Execution  Time. 

If  software  failures  are  tracked  within  testing 
intervals  then  one  of  the  five  following  models  are 
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used:  Generalized  Poisson,  Nonhonogenous  Poisson, 

Brooks  and  Motley,  ‘Schneidwind,  or  the  S-Shaped 
Reliability  Growth.  SMERF5  is  free  to  any  government 
agency  or  private  company  (a  token  fee  may  be  required 
for  materials  and  handling.) 

ff.  Tool.  TCAT  and  TCXT-FAIH 

(1)  Metric(s)  supported.  Depth  of  testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Provides  output  in 
ASCII  file. 

(4)  Platform (s) /language (s)  supported. 

(a)  TCAT:  UNIX,  MS-DOS,  OS/2,  AIX,  XENIX  / 

C,  C++,  Ada,  COBOL,  FORTRAN. 

(b)  TCAT-PATH:  UNIX,  MS-DOS,  OS/2,  AIX, 

XENIX  /  C. 

(5)  Vendor  address/telephone. 

Software  Research  Inc. 

625  Third  St. 

San  Francisco,  CA  94107-1997 
1(800)  942-7638 

(6)  Vendor  supplied  information.  TCAT  measures 
structural  completeness  at  the  module  level  using  the 
logical  segment,  or  "Cl"  metric,  employing  source 
instrumentation  techniques.  TCAT  is  a  powerful  aid  to 
unit  and  small  system  testing.  Coverage  reports  show 
segment  hit,  not-hit,  cumulative  percentages,  and 
histograms.  TCAT-PATH  measures  module  level  test 
coverage  at  path  level  using  source  instrumentation 
techniques.  An  algorithm  generates  set  of  all  possible 
execution  paths. 

gg.  Tool.  TESTBED 

(1)  Metric(s}  supported.  Complexity,  depth  of 
testing. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  report.  Output  provided  in 
standard  interface  file. 

(4)  Platform (s) /language (s)  supported.  All 
Major  Platforms  /  Ada,  C,  Pascal,  FORTRAN,  PL/1,  PL/M, 
COBOL. 

(5)  Vendor  address/telephone. 

Program  Analysers 

56  Northbrook  Street 
Newbury,  Berkshire 
UK,  RG13  IAN 
011-44-635-52-8828 

(6)  Vendor  supplied  information.  TESTBED 
comprises  both  Static  Analysis  and  Dynamic  Analysis 
modules.  In  the  static  domain,  source  code  is  analyzed 
to  give  information  on  control  flow,  logical 
complexity,  data  flow  and  variable  usage.  In  the 
dynamic  domain,  the  execution  of  an  instrumented 
version  of  the  source  code  is  monitored  to  provide 
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coverage  netrics  and  report  violations  of  assertions. 
The  information  obtained  is  presented  in  a  series  of 
report  files. 

hh.  Tool.  Tracker /3 000 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  .  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output. 

(4)  Platform(s) /language (s)  supported. 

(5)  Vendor  address /telephone. 

GBS  Consultants  Inc. 

6087  South  Quebec,  Suite  101 
Englewood,  CO  80111 
(303)  721-077 

(6)  Vendor  supplied  information.  Tracker/3000 
offers  management  controls  within  the  work  unit, 
distribution  of  work  loads,  prioritization  of  end  user 
requests  and  problems,  measurement  of  individual  and 
departmental  performance  standards,  equipment 
inventory, maintenance  records  and  service  call  status. 
Tracker/3000  also  offers  a  hardware  and  software 
inventory  tracking  system. 

ii.  Tool.  Ultra  Planner 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output. 

(4)  Platform (s) /language (s)  supported. 

(5)  Vendor  address /telephone. 

Productivity  Solutions 

36  Washington  St.. 

Wellesley,  MA  02181 
(617)  237-1600 

(6)  Vendor  suppliad  information.  Ultra  Planner 
can  centrally  control  and  monitor  projects  divided  into 
elements,  which  are  distributed  among  various 
individuals,  and  roll  up  those  project  segments  to  a 
major  project  level.  Ultra  Planner  was  developed  to 
bridge  the  gap  between  easy-to-use  PC-style  products 
and  the  superuser  products.  For  example,  the  user  can 
quickly  develop  a  work  breakdown  structure  that  is 
intuitive  and  requires  no  restrictive  coding  scheme. 
Project  data  can  reside  on  multiple  computers,  yet  the 
information  can  be  viewed  and/or  modified  in  real  time, 
and  in  a  seamless  manner. 

jj.  Tool.  VIA/SmartDop 

(1)  Metric(s)  supported.  Complexity. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table (s). 

(3)  Output.  Text  reports.  Updated  version  due 
in  4th  Quarter  FY92  will  provide  output  in  ASCII  file. 

(4)  Platform (s)/ language (s)  supported.  MVS: 

COBOL. 
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(5)  Vendor  address/ telephone. 

VIASOFT 

3033  North  44th  Street 
Phoenix,  Arizona  85018 
l-(800)525-7775 

(6)  .  Vendor  supplied  information.  VIA/SmartDoc 
is  a  static  analysis  and  documentation  generator  that 
provides  IS  professionals  with  comprehensive 
information  about  COBOL  programs  on  demand.  This 
includes  a  wide  array  of  documentation  and  program 
level  metrics  besides  those  detailed  above. 

)ck.  Tool.  Viewpoint 

(1)  Metric (s)  supported.  Cost,  schedule. 

(2)  Data  elements  subject  to  automation:  See 
corresponding  candidate  metric  tool  table(s) . 

( 3 )  Output . 

(4)  Platform (s) /language (s)  supported. 

(5)  Vendor  address/telephone. 

Computer  Aided  Management  Inc. 

1318  Redwood  Kay,  Suite  210 
Petaluma,  Ca 
(800)  635-5621 

(6)  Vendor  supplied  information.  Viewpoint  is  a 
cost-scheduling  utility  for  its  Viewpoint  high-end 
project-management  program.  ViewPoint/CSI,  automates 
data  transfer  between  schedule  data  in  Viewpoint  and 
cost  data  in  M*PM,  the  cost-schedule  control  system 
from  Micro-Frame  Technologies  Inc. ,  thus  eliminating 
the  need  to  rekey  data. 

D-2  0.  Tool  candidate  s\UBmary 

a.  Table  D-12  identifies  all  the  tools  that  were 
reviewed  to  prepare  this  appendix.  The  name  and 
telephone  number  of  each  product's  vendor  is  supplied. 
The  status  column  of  the  table  represents  the  results 
of  the  review.  There  are  three  categories  of  findings. 

(1)  Failed  criteria.  The  tool  was  unable  to 
meet  the  criteria  stated  in  paragraph  0-2  for  any  of 
the  metrics. 

(2)  Failed  to  respond.  The  vendor  failed  to 
respond  to  the  survey  form  or  failed  to  return 
telephone  calls  made  on  behalf  of  the  review. 

(3)  Candidate.  The  tool  satisfies  the  criteria 
of  paragraph  D-2.  This  in  no  way  represents  an 
endorsement  of  the  tool  in  terms  of  its  functionality, 
cost,  usability  or  any  other  factor  other  than  those 
specified  in  paragraph  D-2. 

b.  The  target  metrics  column  of  Table  D-12 
identifies  the  metrics  for  which  the  tool  was  examined. 
It  is  possible  that  the  tool  may  address  other  metrics 
as  well. 

c.  Figure  D-1  graphically  represents  the  overall 
results  of  the  metrics  tool  candidate  preliminary 
survey . 
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^Abla  D-12*  Matrie  Tool  Candidata  Suaiaary 


ADAMAT 


AdaCount 


AdaQuest 


AdaRAIO 


AdaTune 


Candidate 


Candidate 


McCabe  Aaaoc.lnc. 


Dynamic  teaearch 


Failed  Criteria  Ialsys  Inc. 


Candidate 


Candidate 


Candidate 


General  Keteareh 


Proprietary  SW 
Systems 


AISYS  Inc. 


FIT  ADAS 


Artemis 

Candidate 

AutoFlow-C 

Candidate 

Failed  to  Respond  Mark  V  Systems  ltd. 


Failed  to  Respond  SoftwareSys terns* 
Design 


Failed  to  Respond  Research  Triangle 


luces  Management 


Basis  Branch  Anal 


Battlemap  Anal 


C-DOC 


C-Metric 


CA- Superproject 


CARO too  Is 


CMS-2  Code  Audit 


CHS>2  Test  Cover 


COSTAR 


CAP/PLAY  Tl  for  X 


cf  low 


DSB  BOT  CCC 


DSI  I  BOT  Checkpoint 


CST  -  COST 
SCH  •  SCHEDULE 

CRU  •  COMPUTER  RESOURCE  UTILIZATION 
ROT  -  REQUIREMENTS  TRACEABILITY 
ROS  •  REQUIREMENTS  STABILITY 


Candidate 


Candidate 


Software  Blacksmith 


CofTputer  Associates 


Failed  to  Respond  CARDtools  Systems 


Failed  to  Respond  Concurrent  Computer 


Candidate 


Candidate 


Fleet  Combat  Dir, 


Fleet  Combat  Dir, 


Fai led  Criteria  I  Softstar  Systems 


Failed  Criteria 


Failed  to  Respond  I  Digital  Equipment 


Failed  Criteria  I  Softool  Corp 


Candidate 


Software  Product¬ 
ivity  Research 


DSB  •  DESIGN  STABILITY 
CPX  •  COMPLEXITY 
BOT  -  BREADTH  OF  TESTING 
DOT  -  DEPTH  OF  TESTING 
FIT  •  FAULT  PROFILES 
RLY  -  RELIABILITY 


TELEPHONE 


AutoCASE  Technology 


Failed  to  Respond  Hewlett-Packard 


Failed  Criteria  I  McCabe  Assoc.  Inc. 


Failed  Criteria  I  Software  Blacksmith 


301-596*3060 


508-475-9090 


617-270-0030 


805-964-7724 


213-394-5233 


617-270-0030 


818-995-7671 


714-625-6147 


919-541-6629 


703-222-1111 


408-446-2273 


800-538-8787 


301-596-3080 


416-858-4466 


416-856-4466 


800-237-9273 


408-559-4240 


800-631-2154 


619-553-9446 


619-553-9446 


603-672-0987 


415-957-1441 


800-332-4636 


805-683-5777 


617-273-0140 
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Tabla  D-12.  Matrle  Tool  Casdidata  Summary  (Cost'd.) 


TARGET  METRICS 


CRX  DOT 


TOOL 


CodtMap 


ROT  RQS  DSB  BOT  Config  Mgmt  Fac 


RQT  RQS  DSB  BOT  Oclti/IOOO 


CRX  Design  Cpx  Tool 


ROT  ROS  DSB  BOT  DocGen 


ROT  RQS  DSB  BOT  EPOS 


ROT 


DOT 


CPX  DOT 


ROT 


CPX  DOT 


CPX  DOT 


CRU 


DOT 


CRU 


CPX  DOT 


CST  SCH 


BOT  I  Exctlerator 


FORTRAN  Test  Ins 


FPXpert 


Gaia 


HP  64000  UX  Micro 


HP  AxA(^8  Prg  Supp 


HP  GlancePlus  UX 


Hindsight 


J73AVS  B  RXVP80 


Laser  RX/UX 


Logiscope 


Lotus  123/Project 


ROT  ROS  DSB  BOT  Maestro  It 


FLT  I  I  I  IMIL/SOPTOUAL 


CST  SCH 


CST  SCH 


CPX*  DOT 


MicroPlanner 


Multftrak 


PC/UX/VX  Metric 


CST  •  COST 
SCH  -  SCHEDULE 

CRU  -  COMPUTER  RESOURCE  UTILIZATION 
ROT  •  REQUIREMENTS  TRACEABILITY 
RQS  -  REQUIREMENTS  STABILITY 


STATUS 

VENDOR 

TELEPHONE 

Candidate 

CADRE  Technology 

406-562-0106 

Failed  to  Respond 

Expertware  Inc* 

406-746-0706 

Candidate 

OualTrak  Corp. 

A08-730-2674 

Candidate 

Digital  Equipment 

600-332-4636 

Failed  to  Respond 

Corporate  Computer 
Systeni 

908-946-3800 

Failed  to  Respond 

McCabe  Assoc.  Inc. 

301-596-3060 

Failed  to  Respond 

SW  Systems  Design 

714-625-6147 

Failed  to  Respond 

SPS  SW  Products  B 
Services 

212-666-3790 

Failed  to  Respond 

Intersolv 

301-230-3200 

Failed  Criteria 

Softool  Corp. 

605-663-5777 

Failed  Criteria 

Howard  Rubin  Assoc. 

617-661-7171 

Failed  Criteria 

Honeywell  Inc. 

612-762-7327 

Failed  Criteria 

Hewlett-Packard 

601-974-1700 

Failed  Criteria 

Hewlett-Packard 

601-974-1700 

Candidate 

Hewlett-Packard 

600-637-7740 

Candidate 

Advance  Software  | 

Automation 

408-492-1668 

Candidate 

General  Research 

605-964-7724 

Candidate 

Hewlett-Packard 

600-637-7740 

Candidate 

Verilog,  Inc. 

600-424-3095 

Candidate 

Convergics  Corp.  i 

619-669-2433 

Failed  Criteria 

Softlab,  Inc. 

201-239-3666 

Candidate 

The  Carman  Group 

214-667-5069 

Candidate 

Micro  Planning  Int. 

415-389-1420 

Candidate 

Multi trak  Software 
Development  Corp. 

617-462-6677 

Candidate 

Set  Laboratories 

503-629-7123 

DSB  •  DESIGN  STABILITY 

CPX  •  COMPLEXITY 

BOT  •  BREADTH  OP  TESTING 

DOT  •  DEPTH  OF  TESTING 

FLT  •  FAULT  PROFILES 

RLY  •  RELIABILITY 
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TARGET  METRICS 


CST  SCH 


ROT 

CPX 

CST 

SCH 

CPX  DOT 


ROT  RQS  DSB  BOT 


ROT  I  RQS  DSB  lOT 


CST  SCH 


ROT  RQS  DSB  SOT 


ROT  RQS  DSB  BOT 


CPX  DOT 


ROT 


DOT 


CST 


CPX  DOT 


ROT  RQS  DSB  i  BOT 


Primtvera 


ProMod  Plus 


Project  Bchtdular 


OA  C  i  OA  FORTRAN 


R-Tr»ce 


REVIC 


RT  Tool 


RTM 


S-TCAT 


STRIPS 


SW  Anal  Test  Tool 


SW  Cost  Est  Model 


SW  Dual  Mgmt  Sys 


SW  Thru  Pictures 


Safe  C  Runtime 


Stat.  Model  I  Est 
Rely  Func  (SMERF) 


START 


Struct.  Archt. 


CPX 


ROT 


ROT  DOT  BOT 


CPX  DOT  Ir-Seope 


DOT  TAOS 


ROT  ROS  DSB  BOT  TAGS  Case  2  Tool 


DOT 


CM*  DOT 


-  COST 

•  SCHEDULE 

•  COMPUTER  RESOURCE  UTILIZATION 

•  REQUIREMENTS  TRACEABILITY 

•  REQUIREMENTS  STABILITY 


Candidate 


Failed  to  Respond 


Candidate 


Candidate 


Candidate 


Failed  to  Respond 


Failed  Criteria 


Failed  to  Respond 


Candidate 


Failed  Criteria 


Failed  Criteria 


Failed  Criteria 


Failed  Criteria 


Failed  Criteria 


Failed  to  Respond 


Failed  Criteria 


Candidate 


Failed  Criteria 


Failed  to  Respond 


Failed  to  Respond 


Failed  Criteria 


Failed  to  Respond 


Failed  to  Respond 


TCAT  A  TCAT-PATH 

Candidate 

TESTBED 

Candidate 

Priaiavera  Systems 


Meridian  Software 


Sc i tor  Corp. 


PrograrnninQ  Research 


Protocol 


Ascent  Logic  Corp. 


Air  Force  Cost  Anal. 


Teledyne  Brown 


Marconi  Systems 


Software  Research 


AMS 


IBM  Corp 


NASA 


SW  Quality  Tools 


Interactive  Develop 
•a»nt  Environment 


Blossom/Catalytix 


Naval  Surface 
Warfare  Center 


McCabe  Assoc.  Inc. 


LBMS 


Programming 
Envi  ronments 


Software  Research 


UNISYS 


Teladyne  Brown 


Software  Research 


Program  Analyzers 


DSB  •  DESIGN  STABILITY 
CPX  •  COMPLEXITY 
BOT  •  BREADTH  OF  TESTING 
DOT  -  DEPTH  OF  TESTING 
FLY.-  FAULT  PROFILES 
RLY  -  RELIABILITY 
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TARC 

;et  h 

ETRIC 

S 

TOOL 

STATUS 

VENDOR 

TELEPHONE 

ROT 

ROS 

DSB 

JOT 

Tt«TMork/Ad« 

Failed  to  Respond 

CADRE  Technology 

40V35V59S0 

ROT 

ROS 

DSB 

BOT 

Toolset  2167a 

Failed  Criteria 

Adv  Sys  Technology 

S01-261>086Z 

CST 

SCH 

Tracker  3000 

Candidate 

6BS  Consultants 

303-721-0777 

CRX 

IWISET 

Failed  to  Respond 

UNISYS 

313-972-7000 

CST 

SCH 

UltraPlenner 

Candidate 

Productivity 

Solutions 

617>237-1600 

ROT 

ROS 

DSB 

BOT 

VASTT 

Failed  Criteria 

Vitro  Corp. 

301-231-13SB 

DOT 

VAX  PCA 

Failed  to  Respond 

Digital  Equipment 

e00*33Z*A6M 

DOT 

VAX set 

Failed  to  Respond 

Digital  Equipment 

600-332-4636 

ZPt 

VlA/SmsrtOoc 

Candidate 

VIASOFT 

6O2-95Z-00B0 

DOT 

VIA/SmsrtTest 

Failed  Criteria 

VIASOFT 

602-952-0050 

ROT 

ROS 

DSB 

BOT 

VISTA 

Failed  Criteria 

Vitro  Corp 

301 -231 -1358 

CST 

SCH 

VS£N 

Failed  Criteria 

Vitro  Corp 

301'23M35S 

CST 

SCH 

Viewpoint 

Candidate 

Computer  Aided  Mgmt 

600-635-5621 

CST  •  COST 

SCH  -  SCHEDULE 

CRU  •  COMPUTER  RESOURCE  UTILIZATION 
ROT  •  REQUIREMENTS  TRACEABILITY 

ROS  •  REQUIREMENTS  STABILITY 

DSB  *  DESIGN  STABILITY 

CPX  •  COMPLEXITY 

BOT  •  BREADTH  OF  TESTING 

DOT  •  DEPTH  OF  TESTING 

FLT  •  FAULT  PROFILES 

RLY  •  RELIABILITY 
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NUMBER  OF  TOOLS  SURVEYED^ 


SURVEY  OUTCOME 

■  candidates  0  possible  candidates  □  rejected* 

I'fOOLS  MAY  OUAJFY  roe  MULTTUE  METWCS 

*  cenm*.  metwc  cmta  el£wb<t8  cookto  h  roerrA«i  morm 


Figura  D-l.  Caadidata  Tools  for  Matries  Cellaetioa 
and  Raporting 
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Glossary 
8aetlon  Z 

Texina  and  Dafinitions 
Alloeatad  Jasalina 

The  initial,  approved  performance  oriented 
specification  governing  the  development  of 
configuration  items  (CIs)  that  are  part  higher 

level  Cl,  in  which  each  specification  (a)  defines  the 
functional  characteristics  that  are  allocated  from 
those  of  the  higher  level  Cl,  (b)  establishes  the  tests 
required  to  demonstrate  achievement  of  its  allocated 
functional  characteristics,  (c)  delineates  necessary 
interface  requirements  with  other  associated 
configuration  items,  and  (d)  establishes  design^ 
constraints,  if  any,  such  as  component  standardization, 
use  of  inventory  items  and  integrated  logistics  support 
requirements.  (Reference  DOD-STD-480A) 

Automated  laformatioB  8ystem  (AZ8) 

A  combination  of  information,  computer  and 
telecommunications  resources  and  other  information 
technology  and  personnel  resources  that  collects, 
records,  processes,  stores,  communicates,  retrieves, 
and  displays  information.  (Reference  DOD-STD-7935A) 

Baseline 

A  configuration  identification  document  or  a  set  of^ 
documents  formally  designated  and  fixed  at  a  specific 
time  during  a  configuration  item's  life  cycle. 

Baseline,  plus  approved  changes  from  those  baselines 
constitute  the  current  configuration.  (Reference  DOD- 
STD-480A) 

Benchmark  Test  Files  (BMTF) 

A  data  base  of  known  content  against  which  a  controlled 
set  of  inputs  is  processed  and  from  which  output 
results  may  be  predicted.  This  term  is  used  in 
reference  to  a  test  environment  and  pre**established 
test  cases/data. 

CA8B  Tools 

Systems  for  building  systems;  automates  the 
requirements  analysis,  design  and  development  process. 

code  Walk-through 

The  process  of  assessing  the  level  of  software 
performance  and  design  structure  that  requires  the 
developer  to  demonstrate  the  capabilities  of  the 
software  to  technical,  functional,  and  user 
representatives. 


( 
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Computar  Raaoureas 

The  totality  of  computer  personnel,  documentation, 
services,  and  supplies  applied  to  a  given  effort. 

This  includes  hardware,  software,  services,  personnel, 
documentation  and  supplies.  (Reference  DOD-STD-2167A) 

Computer  Raaoureas  Nanaganant  Plan  (CRMP)  or  Computar 
Resource  Life  Cycle  Kanagamant  Plan  (CRLCMP) 

The  primary  planning  document  to  be  used  at  all 
decision  levels  for  assessing  the  adequacy  of  the 
overall  computer  resources  management  efforts 
throughout  the  system  life  cycle.  Used  only  for  AR  70- 
1  developed  systems.  (Reference  AR  70-1,  DODI  5000.2) 

Computer  Raaoureas  Woric  Group  (CRW6) 

Established  by  the  Material  Developer  after  Milestone  I 
for  each  AR  70-1  system  to  aid  in  the  management  of 
system  computer  resources.  The  CRWG  assists  in 
insuring  compliance  with  policy,  procedures,  plans  and 
standards  established  for  computer  resources. 

Computar  Softvara  Componant  (CSC) 

A  distinct  part  of  a  computer  software  configuration 
item  (CSCI) .  CSCs  may  be  further  decomposed  into  other 
CSCs  and  Computer  Software  Units  (CSUs) .  (Reference 
DOD-STD-2167A) 

Computer  Softvara  Configuration  Itam  (CSC!) 

An  aggregation  of  software  or  any  of  its  discrete 
portions,  which  satisfies  an  end  use  function  and  is 
designated  by  the  Government  for  Configuration 
Management.  (Reference  DOD-STD-2167A) 

Computer  Softvara  Unit  (CSU) 

An  element  specified  in  the  design  of  a  Computer 
Software  Component  (CSC)  that  is  separately  testable. 
(Reference  DOD-STD-2167A) 

Configuration  Item  (CZ) 

An  aggregation  of  hardware/ software,  or  any  of  its 
discrete  portions,  which  satisfy  an  end  use  function 
and  is  designated  by  the  Government  for  configuration 
management.  (Reference  DOD-STD-480A) 

Configuration  Management 

A  discipline  applying  technical  and  administrative 
direction  and  surveillance  to  (a)  identify  and  document 
the  functional  and  physical  characteristics  of  a 
configuration  item,  (b)  control  changes  to  those 
characteristics,  and  (c)  record  and  report  change 
processing  and  Implementation  status.  (Reference  DOD- 
STD-480A) 
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Critical  Daaiga  Raviav  (CDR) 

A  review  conducted  ’to  determine  that  the  detail  design 
satisfies  the  performance  and  engineering  requirements 
of  the  development  specification;  to  establish  the 
detail  design  compatibility  among  the  item  and  other 
items  of  equipment,  facilities,  computer  programs,  and 
personnel;  to  assess  producibility  and  risk  areas;  and 
to  review  the  preliminary  product  specification. 
(Reference  MIL-STD-1521B) 

Cyela/systam  Test 

The  final  phase  of  developer's  testing  which  involves 
the  testing  of  modules/programs/cycles  which  are 
integrated  into  the  total  system. 

Davalopar  Tests 

Testing,  modeling,  and  experimentation  conducted  by  the 
system  developer.  Formal  tests  normally  involve  system 
level  integration  and  certification  by  the  developer 
with  formal  government  monitoring.  Informal  tests 
involve  lower  level  code  and  unit  development  with 
internal  integration  between  system  elements. 
Experimentation  includes  a  wide  variety  of  tests, 
models,  development  techniques  and  simulations  used  to 
validate  design  concepts  and  theories. 

Developmental  Tests  (DT) 

Tests  usually  conducted  by  an  organization  independent 
of  the  developer (s)  in  order  to  validate  total  system 
conformance  to  technical  and  functional  specifications 
and  ensure  the  system  is  ready  for  formal  or  limited 
user  testing.  Formal  tests  focus  primarily  on  total 
systems  integration.  Limited  tests  are  used  primarily 
with  priority  1,  2,  or  3  changes  during  PDSS  and  focus 
on  the  specific  changes  being  made. 

Development  Tools 

Products  which  are  necessary  to  prepare,  test  and 
evaluate  software  units  currently  under  development. 

Driver 

Software  which  performs  two  functions:  control  of  an 
external  hardware  device  or  controlling  the  execution 
of  other  programs. 

Dynamic  Analysis 

Involves  execution  or  simulation  of  a  development  phase 
product.  It  detects  errors  by  analyzing  the  response 
of  a  product  to  sets  of  input  data. 

Emulation 

An  interpretation  similar  to  simulation,  however,  the 
interpretation  is  done  through  hardware  or  microcode  or 
the  process  of  using  software  or  peripherals  to  make 
one  set  of  hardware  operate  like  another. 
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Enginaariag  Chaaga  Proposal  -  Softvara  (ECP-8) 

A  term  which  includes  both  a  proposed  engineering 
change  and  the  documentation  by  which  the  change  is 
described  and  suggested.  DA  Form  5005-R  is  used  to 
document  proposed  changes  to  software  baselines  and 
associated  baseline  documentation.  (Reference  DOD-STD- 
480A) 

Firmware 

A  combination  of  a  hardware  device  and  computer 
instructions  or  computer  data  that  reside  as  read-only 
software  on  the  hardware  device.  The  software  cannot 
be  readily  modified  under  program  control.  (Reference 
DOD-STD-2167A) 

Formal  Qualification  Tasting  (FQT) 

A  process  that  allows  the  contracting  agency  to 
determine  whether  a  configuration  item  (subsystem) 
complies  with  the  allocated  requirements  for  that  item. 
(Reference  DOD-STD-2167A) 

Functional  Baseline 

The  initial,  approved  technical  documentation  for  a 
configuration  item  which  prescribes  (a)  all  necessary 
functional  characteristics,  (b)  the  tests  required  to 
demonstrate  achievement  of  specified  functional 
characteristics,  (c)  the  necessary  interface 
characteristics  with  associated  CIs,  (d)  the  Cl’s  key 
functional  characteristics  and  its  key  lower  level  CIs, 
if  any,  and  (e)  design  constraints,  such  as,  envelope 
dimensions,  component  standardization,  use  of  inventory 
items,  and  integrated  logistics  support  policies. 
(Reference  DOD-STD-480A) 

Functional  Configuration  Audit  (FCA) 

A  formal  audit  to  validate  that  the  development  of  a 
configuration  item  has  been  completed  satisfactorily 
and  that  the  configuration  item  has  achieved  the 
performance  and  functional  characteristics  specified  in 
the  functional  or  allocated  configuration 
identification.  In  addition,  the  complete  operation 
and  support  docximents  are  required.  (Reference  MIL-STD- 
1521B) 

Implementation  Procedures  (IP) 

A  document  developed  in  accordance  with  DOD-STD-7935A 
which  provides  information  to  the  user  and  data 
processing  personnel  to  install  the  AIS  and  achieve 
operational  status. 

Independent  Verification  and  Validation  (IV&V) 
Verification  and  validation  performed  by  a  contractor 
or  Government  agency  that  is  not  responsible  for 
developing  the  product  or  performing  the  activity  being 
evaluated.  IV&V  is  tailored  to  the  risk  associated 
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with  the  system.  IV&V  is  conducted  separately  from  the 
software  development  activities.  (Reference  DOD-STD- 
2167A) 

Zstarfaca 

The  process  of  passing  data  between  systems. 

Interim  Change  Package  (ICP) 

A  software  modification  release  of  ECP-s  which,  because 
of  urgency,  regulatory  requirement  or  special  need, 
must  be  provided  before  the  availability  of  the  next 
scheduled  Software  Change  Package. 

Interoperability  . 

The  ability  of  systems,  units,  or  forces  to  provide 
services  to  and  accept  services  from  other  systems, 
units  or  forces  and  to  use  the  services  to  enable  them 
to  operate  effectively  together. 

Issues  and  Criteria 

Issues  are  questions,  the  answers  to  which  permit  the 
overall  system  evaluation.  Criteria  are  the 
quantitative  or  qualitative  standards  by  which  issues 
are  evaluated. 

Lead-Site  Verification  Test  (L8VT) 

A  system  test  in  a  selected  operational  environment 
where  an  Interim  System  Change  Package  (ICP)  is 
evaluated  to  determine  if  it  meets  the  ICP's  goals  and 
objectives. 

Left-of-Baseline  (LOB) 

The  manual  and  automated  processes  of  extracting 
selected  data  and  reducing  them  to  input  file  and 
transaction  formats  acceptable  for  building  or 
initializing  a  data  base  for  a  new  system.  Normally 
associated  with  conversion  requirements  or  parallel 
testing. 

Materiel  Systems  Computer  Resources  (M8CR) 

Computer  resources  acquired  for  use  as  integral  parts 
of  weapons;  command  and  control;  communications; 
intelligence  and  other  tactical  or  strategic  systems 
and  their  support  systems.  The  term  also  includes  all 
computer  resources  associated  with  specific  program 
developmental  TiE,  operational  testing,  and  post 
deployment  software  support  including  weapon  system 
training  devices,  automatic  test  equipment,  land-based 
test  sites,  and  system  integration  and  test 
environments . 


Ketri;iS 

A  quantitative  value,  procedure,  methodology,  and/or 
technique  which  allows  one  the  ability  to  measure 
various  aspects  and  characteristics  of  software. 
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Mondavalopmantal  Xtaa  (NDI) 

Deliverable  items  that  are  not  developed  under  the 
contract  but  are  provided  by  the  contractor,  the 
Government  or  a  third  party.  NOI  may  be  referred  to  as 
reusable.  Government  furnished,  or  commonly  available 
software,  Jiardware  or  total  systems,  depending  upon  the 
source.  (Reference  DOD-STD-2167A) 

Parallel  Tasting 

Testing  that  demonstrates  whether  or  not  two  versions 
of  the  same  application  are  consistent,  or  two  systems 
performing  the  same  function. 

Physical  Configuration  Audit  (PCX) 

The  technical  examination  of  a  designated  configuration 
item  to  verify  that  the  configuration  item  •‘as-built" 
conforms  to  the  technical  documentation  which  defined 
the  Cl.  (Reference  MIL-STD-1521B) 

Preliminary  Design  Review  (PDR) 

This  review  is  conducted  for  each  configuration  item  or 
aggregate  of  configuration  items  to  (1)  evaluate  the 
progress,  technical  adequacy,  and  risk  resolution  (on  a 
technical,  cost,  and  schedule  basis)  of  the  selected 
design  approach,  (2)  determine  its  compatibility  with 
performance  and  engineering  specialty  recjuirements  of 
the  Hardware  Configuration  Item  (HWCI)  development 
specification,  (3)  evaluate  the  degree  of  definition 
and  assess  the  technical  risk  associated  with  the 
selected  manufacturing  compatibility  of  the  physical 
and  functional  interfaces  among  the  configuration  item 
and  other  items  of  equipment,  facilities,  computer 
software,  and  personnel.  For  CSCIs,  this  review 
focuses  on:  (1)  the  evaluation  of  the  progress, 
consistency,  and  technical  adequacy  of  the  selected 
top-level  design  and  test  approach,  (2)  compatibility 
between  software  requirements  and  preliminary  design, 
and  (3)  on  the  preliminary  version  of  the  operation  and 
support  documents.  (Reference  MIL-STD-1521B) 

Program 

A  separately  compilable,  structural  (closed)  set  of 
instructions  most  precisely  associated  with  early 
generations  of  computers.  Synonymous  with  computer 
program.  Contrast  with  Software  Unit.  (Reference  DOD- 
STD-7935A) 

Program  Folder 

A  repository  collecting  material  pertinent  to  the 
development  or  support  of  software.  Contents  typically 
include  (either  directly  or  by  reference) ,  design 
considerations  and  constraints,  design  documentation 
and  data,  schedule  and  status  information,  test 
requirements,  test  cases,  test  procedures  and  test 
results. 
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Racovary/Raeonfiguration  Tasting 

Testing  that  verifies  the  recovery  process  and 

component  parts'  effectiveness.  It  validates  that 

enough  backup  data  is  preserved  and  stored  in  a  secure 

location. 

Regression  Testing 

Testing  of  a  computer  program  to  assure  correct 
performance  after  changes  were  made  to  code  that 
previously  performed  correctly,  or  the  process  entails 
testing  or  retesting  those  areas/aspects  of  a  system 
which  will  or  could  be  affected  by  a  change. 

Ralaesa 

A  configuration  management  action  whereby  a  particular 
version  of  software  is  made  available  for  a  specific 
purpose  (e.g. ,  released  for  test) .  (Reference  DOD-STD- 
2167A) 

Required  Operational  Characteristics 
Qualitative  and  quantitative  system  performance 
parameters,  proposed  by  the  user  and  approved  by  the 
Army,  that  are  primary  indicators  of  a  system's 
capability  to  accomplish  its  mission  (operational 
effectiveness)  and  to  be  supported  (operational 
suitability) .  Required  operational  characteristics 
are  usually  tested  and  evaluated  by  operational  testing 
and  evaluation  to  ascertain  achievement  of  approved 
goals  and  thresholds  for  these  characteristics. 

Required  Technical  Characteristics 

Quantitative  system  performance  parameters  approved  by 
the  Army  management  that  are  selected  as  primary 
indicators  of  technical  achievement.  These  might  not  be 
direct  measures  of,  but  always  should  relate  to  a 
system's  capability  to  perform  its  required  mission 
function  and  to  be  supported.  Required  technical 
characteristics  usually  are  tested  and  evaluated  to 
ascertain  approval  goals  and  thresholds  for  these 
characteristics . 

Requirements  Trace 

Assuring  requirements  flow  from  the  user  specifications 
through  design  and  Implementation  of  the  product. 

Right-ef-Baseline  (ROB) 

The  automated  process  of  building  a  data  base  from  LOB 
products,  or  the  initialization  of  new  files  Introduced 
for  the  first  time.  Normally  associated  with 
conversion  requirements  or  parallel  testing. 

Simulation 

The  process  of  conducting  experiments  with  a  model  for 
the  purpose  of  understanding  the  behavior  of  the 
system.  Simulations  may  be  dynamic,  engineering 
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(scientific) r  environmental,  instruction-level, 
statement- level,  and  system  level.  For  AIS,  simulation 
entails  summary  files  to  simulate  an  internal  or 
external  interface  input.  (Reference  DODD  5000.3-M-l) 

Software  Acceptance  Test  (SAT) 

A  operational  test  of  a  new  system  or  changes  to  a 
deployed  system,  directed  by  an  independent  tester  and 
conducted  in  a  field  environment  using  a  production 
data  base  and  executed  on  target  hardware. 

Software  Change  Paclcage 

One  or  more  changes  which  have  been  approved  and 
scheduled  for  implementation,  as  a  group,  by  the 
appropriate  configuration  control  board  lAW  DOD-STD- 
480A. 

Software  Development  Folder  (SDF) 

A  repository  for  a  collection  of  material  pertinent  to 
the  development  or  support  of  software.  Contents 
typically  include  (either  directly  or  by  reference) , 
design  considerations  and  constraints,  design 
documentation  and  data,  schedule  and  status 
information,  test  requirements,  test  cases,  test 
procedures  and  test  results.  (Reference  DOD-STD-2167A) 

Software  Development  Library  (SDL) 

A  controlled  collection  of  software,  docvimentation,  and 
associated  tools  and  procedures  used  to  facilitate  the 
orderly  development  and  subsequent  support  of  software. 
The  SDL  includes  the  developmental  configuration  as 
part  of  its  contents.  A  software  development  library 
provides  storage  of  and  controlled  access  to  software 
and  documentation  in  human-readable  form, 
machine-readable  form,  or  both.  The  library  may  also 
contain  management  data  pertinent  to  the  software 
development  project.  (Reference  DOD-STD-2167A) 

Software  Engineering  Environment  (SEE) 

Set  of  automated  tools,  firmware  devices,  and  hardware 
necessary  to  perform  the  software  engineering  effort. 
The  automated  tools  may  include  but  are  not  limited  to 
compilers,  assemblers,  linkers,  loaders,  operating 
systems,  debuggers,  simulators,  emulators,  test  tools, 
documentation  tools,  and  data  base  management  systems. 
(Reference  D0D-STD-2167A) 

Software  Qualification  Test  (8QT) 

Independent  developmental  test  conducted  on  target 
hardware,  but  normally  not  in  an  operational 
environment. 

software  Specification  Review  (SSR) 

A  review  of  the  finalized  software  configuration  item 
requirements  and  operational  concept.  The  SSR  is 
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conducted  when  software  requirements  have  been 
sufficiently  defined  to  evaluate  the  developer's 
responsiveness  to  and  interpretation  of  the  system, 
segment  or  prime  item  level  requirements.  (Reference 
MIL-STD-1521B) 

software  feat  Eavironment 

A  set  of  automated  tools,  firmware  devices,  and 
hardware  necessary  to  test  software.  The  automated 
tools  may  Include  but  are  not  limited  to  test  tools 
such  as  simulation  software,  code  analyzers,  etc.,  and 
may  also  include  those  tools  used  in  the  software 
engineering  environment  (SEE) .  (Reference  DOD-STD- 
2167A) 

Software  Unit 

Any  logical  set  or  groupings  of  instructions  to  a 
computer,  such  as  a  module  or  package.  (Reference  DOD- 
STD-7935A) 

Statement  of  Worlc  (SOW) 

A  statement  of  contract  requirements  that  is  used  for 
defining  and  achieving  program  goals.  The  SOW  provides 
the  basic  framework  for  a  particular  effort.  It  is  a 
document  by  which  all  nonspecification  requirements  for 
developer  efforts  must  be  established  and  defined 
either  directly  or  with  the  use  of  specific  cited 
documents . 

Static  Analysis 

Direct  analysis  of  the  form  and  structure  of  a  product 
without  executing  the  product.  It  may  be  applied  to 
requirements,  design,  or  code. 

Stress  Test 

A  program  which  exercises  code  up  to,  including  and 
beyond  all  stated  limits  in  order  to  exercise  all 
aspects  of  the  system  (e.g.,  to  include  hardware, 
software,  and  communications).  The  purpose  is  to 
insure  that  response  times  and  storage  capacities  meet 
requirements . 

Supplemental  Site  Test 

Testing  conducted  on  multi-vendor  and/or 
multi-operating  system  environment  or  for 
conditions/functions  not  readily  available  at  a  primary 
test  site. 

Supportability 

The  degree  to  which  a  system  can  be  maintained  or 
sustained  in  an  operational  environment. 

System  Change  Package  (SCP) 

A  group  of  modifications  documented  on  ECP-S  which  are 
packaged  and  implemented  during  post  deployment  phase. 
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Systam  Daeision  Papar 

The  primary  docuneilt  used  to  obtain  MAISRC  approval  for 
information  systems. 

System  Design  Raviav  (SDR) 

This  review  is  conducted  to  evaluate  the  optimization, 
correlation,  completeness,  and  ris}cs  associated  with 
the  allocated  technical  requirements.  Also  included  is 
a  summary  review  of  the  system  engineering  process 
which  produced  the  allocated  technical  requirements  and 
of  the  engineering  planning  for  the  next  phase  of 
effort.  This  review  is  conducted  when  the  system 
definition  effort  has  proceeded  to  the  point  where 
system  characteristics  are  defined  and  the 
configuration  items  are  identified.  (Reference  MIL-STD- 
1521B) 

System  Post-Deployment  Review  (SPR) 

A  review  conducted  after  deployment  of  the  initial 
system  to  evaluate  how  well  the  operational  system  is 
satisfying  user  requirements. 

System  Requirements  Review  (SRR) 

The  objective  of  this  review  is  to  ascertain  the 
adequacy  of  efforts  in  defining  system  requirements. 

It  is  conducted  when  a  significant  portion  of  the 
system  functional  requirements  has  been  established. 
(Reference  MIL-STD-1521B) 

System  Specification 

A  system  level  requirements  specification.  A  system 
specification  may  be  a  System/ Segment  Specification, 
Prime  Item  Development  Specification  (PIOS) ,  or 
Critical  Item  Development  Specification  (CIDS) . 
(Reference  DOD-STD-2167A) 

Target  Hardware 

Suite  of  hardware  designated  as  the  operational 
configuration  of  the  system. 

Test  and  Evaluation  Master  Plan  (TEMP) 

The  )cey  management  tool  for  control  of  the  integration 
of  all  T&E  requirements  for  each  acquisition  effort  and 
is  used  by  decision  review  bodies.  (Reference  DOO 
5000. 2-M  and  DOD  7920. 2-M) 

Test  Bed 

A  system  representation  consisting  partially  of  actual 
hardware  and/or  software  and  partially  of  computer 
models  or  prototype  hardware  and/or  software. 

(Reference  DODI  5000.2) 

Test  Condition  Requirement  (TCR) 

Represents  a  specific  testable  criteria  or  sub-function 
of  a  system.  Often  composed  of  further  subordinate 
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levels  of  test  items  and  specific  test  cases.  It  may 
represent  a  test  scenario  or  be  an  element  of  a  larger 
scenario.  ECP-S's  include  test  conditions.  A  test 
condition  requirement  includes  five  elements: 

1.  Input  instructions  references  (i.e.,  user's 
manual  paragraph) . 

2.  Validation  standard/criteria  reference  (i.e.,  FD 
or  regulation) . 

3.  Data  parameters  or  specific  input  data 
requirements  for  underlying  cases. 

4.  Cross  reference  to  other  TCRs  as  appropriate. 

5.  Primary  output  products  to  be  used  for  validation 
(i.e.,  file  dumps,  specific  reports,  screen  inquiries). 

Test  Hoe)ca 

Software  logic  which  is  integrated  into  the  system  in 
order  to  facilitate  extraction  of  data  to  support  test 
and  analysis. 

Test  Integration  Worlclng  Group  (TIWO) 

Established  by  the  program  sponsor  upon  receipt  of  the 
draft  Operational  Requirements  Document  or  Mission 
Needs  Statement.  This  is  the  primary  group  which 
facilitates  integration  of  TiE  requirements  and 
prepares  the  TEMP. 

Test  Readiness  Review  (TRR) 

A  review  conducted  for  each  CSCI  to  determine  whether 
the  software  test  procedures  are  complete  and  to  assure 
that  the  contractor  is  prepared  for  formal  CSCI 
testing.  (Reference  MIL-STD->1521B) 

Unit  Test 

The  lowest  level  developer  test. 

n 

User  or  Operational  Tests 

A  system-level  test  performed  by  a  test  activity 
independent  of  the  developer  or  the  PM.  The  objective 
of  the  OT  is  to  test  the  entirety  of  the  system  and  is 
performed  by  users  in  e^n  operational  environment. 

Validation 

The  process  of  evaluating  software  to  determine 
compliance  with  specified  requirements.  (Reference  MIL- 
STD-2167A) 

Verification 

The  process  of  evaluating  the  products  of  a  given 
software  development  activity  to  determine  correctness 
and  consistency  with  respect  to  the  products  and 
standards  provided  as  an  input  to  that  activity. 
(Reference  DOD-STD-2167A) 
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Pars ion 

An  identified  and  docxunented  body  of  software. 
Modifications  to  a  version  of  software  (resulting  in  a 
new  version)  require  configuration  management  actions 
by  the  contractor,  the  contracting  agency  or  both. 
(Reference  DOD-STD-2167A) 

Version  Description  Document  (VDD) 

A  document  containing  a  description  of  the  software 
version  for  a  specific  CSCI  or  subsystem.  It  contains 
all  corrective  actions  written  and  resolved.  It  may 
contain  loading  instructions  and  other  specific 
information  to  this  software  version. 

Walk-through 

An  informal,  step-by-step  review  of  a  particular 
product  within  the  development  (i.e.,  program  code, 
test  scenario,  functional  design)  which  allows  feedback 
from  other  members  of  the  development  team  to  the 
creator  of  the  particular  product  being  reviewed. 
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Saetion  IZ 
Acronyms 


AAE  .  Army  Acquisition  Executive 

ACAT  ....  Acquisition  Category 

AdalC  . . .  -Ada  Information  Clearinghouse 

ADP  .  Automated  Data  Processing 

AI  .  Artificial  Intelligence 

AIN  .  Army  Interoperability  Network 

AIS  .  Automated  Information  System 

AMC  .....  Army  Materiel  Command 

AMSAA  ...  Army  Materiel  Systems  Analysis  Activity 
AOA  .  Abbreviated  Operational  Assessment 


ASAP  ....  Army  Streamlined  Acquisition  Program 
ASARC  . . .  Army  Systems  Acquisition  Review  Council 
ASIOE  . . .  Associated  Support  Items  of  Equipment 
ATCCS  . . .  Army  Tactical  Command  and  Control  System 

ATE  .  Automated  Test  Equipment 

AT IRS  ...  Army  Test  Incident  Report  System 

BOIP  ....  Basis  of  Issue  Plan 
BMTF  ....  Benchmark  Test  Files 


CAP  .  Corrective  Action  Plan 

CASE  ....  Computer  Aided  Software  Engineering 
CAIS  ....  Common  Ada  Programming  Support  Envi: 

Interface  Set 

CBTDEV  . .  Combat  Developer 

CCB  .  Configuration  Control  Board 

CDA  .  Central  Design  Activity 

CDR  .  Critical  Design  Review 

CDRL  ....  Contract  Data  Requirements  List 

CE  .  Continuous  Evaluation 

CEPT  ....  Concept  Evaluation  Program  Test 

Cl  . Configuration  Item 

CLS  .  Contractor  Logistics  Support 

CM  .  Configuration  Management 

CMP  .  Configuration  Management  Plan 

CMS  .  Configuration  Management  System 

COE  .  Corps  of  Engineers 


COIC  ....  Critical  Operational  Issues  and  Criteria 

COMPUSEC  Computer  Security 

COMSEC  . .  Communication  Security 

CONOPS  . .  Continuity  of  Operations  Plans 

COOP  ....  Continuity  of  Operations  Plans 

CPU  .  Central  Processing  Unit 

CRISD  . . .  Computer  Resources  Integrated  Support 
Document 

CRMP  ....  Computer  Resources  Management  Plan 

CRU  .  Computer  Resource  Utilization 

CRWG  ....  Computer  Resources  Working  Group 

CSC  .  Computer  Software  Component 

CSCI  ....  Computer  Software  Configuration  Item 
CSE  .  Center  for  Software  Engineering 
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CSTA  ....  Combat  Systems  Test  Activity 
CSU  .....  Computer  Software  Unit 

DA .  Department  of  the  Army 

DAA  .  Designated  Accreditation  Authority 

DAB  .  J)ef ense  Acquisition  Board 

DACS  ....  Data  &  Analysis  Center  for  Software 

DAP  .  Defense  Acquisition  Program 

DASD  ....  Direct  Access  Storage  Device 

DCSOP  . . .  Deputy  Chief  of  Staff  Operations  and  Plans 

DE  .  Developmental  Evaluator 

DID  .  Data  Item  Description 

DISC4  ...  Director  of  Information  Systems  Command, 
Control,  Communications,  and  Computers 

DLA  .  Defense  Logistics  Agency 

DOD  .  Department  of  Defense 

DODD  ....  Department  of  Defense  Directive 
DODI  ....  Department  of  Defense  Instruction 
DOIM  ....  Director  of  Information  Management 
DOT&E  ...  Director,  Operational  Test  and  Evaluation 

DP  .  Design  Process 

DS  .  Database  Specification 

DT . Developmental  Test 

DT&E  ....  Developmental  Test  and  Evaluation 
DTP  .  Detailed  Test  Plan 

DTRR  ....  Developmental  Test  Readiness  Review 
DTRS  ....  Developmental  Test  Readiness  Statement 

ECPs  ....  Engineering  Change  Proposals 

ECP-S  . . .  Engineering  Change  Proposal  -  Software 

ELSEC  . . .  Electronic  Security 

EM . End  User  Mannual 

EOA  .  Early  Operational  Assessment 

ETR  .  Expanded  Test  Report 

EUE  .  Early  User  Experimentation 

EUT .  Early  User  Test 

EUTE  ....  Early  User  Test  and  Experimentation 

FCA  .  Functional  Configuration  Audit 

FD  .  Functional  Description 

FDE  .  Force  Development  Experiment 

FDT  .  Force  Development  Test 

FDTE  ....  Force  Development  Test  and  Experimentation 

FOT  .  Follow-on  Operational  Test 

FP  .  Functional  Proponent 

FQT  . . . . .  Formal  Qualification  Test/Testing 

FUE  .  First  Unit  Equipped 

FUED  ....  First  Unit  Equipped  Delivery 

FVT  .  Functional  Verification  Test 

FYTP  ....  Five-Year  Test  Plan 

GSA  .  General  Services  Administration 

HFE  .  Human  Factors  Engineering 

HOL  .  Higher  Order  Language 
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HQDA  ....  Headquarters ,  Department  of  the  Army 


(  HSC  .  Health  Services  Command 

HUC . Human  Use  Committee 

HW . Hardware 

I/O . -Input/Output 

lAP  .  Independent  Assessment  Plan 

lAR  .  Independent  Assessment  Report 

lAW . In  Accordance  With 

I  CP . Interim  Change  Pac)cage 

IDD  .  Interface  Design  Document 

lEB  .  Independent  Evaluation  Briefings 

lEP  .  Independent  Evaluation  Plan 

lER  .  Independent  Evaluation  Report 

ILS  .  Integrated  Logistic  Support 

ILSP  ....  Integrated  Logistic  Support  Plan 
INSCOM  . .  Intelligence  and  Security  Command 

IOC  .  Initial  Operational  Capabilities 

lOE  .  Independent  Operational  Evaluator 

lOT  .  Initial  Operational  Test 

lOT&E  . . .  Initial  Operational  Test  and  Evaluation 
IP .  Implementation  Plan; 

Implementation  Procedures 
IPR  .  In-Process  Review 

IRS  .  Interface  Requirements  Specification 

IS  .  Information  Systems 

f  ISC  .  Information  Systems  Command 

(  ISEC  ....  Information  Systems  Engineering  Command 

ISS  .  Information  System  Security 

IV&V  ....  Independent  Verification  and  Validation 

JCL  .  Job  Control  Language 

JT . Joint  Test 


LCSEC  . . .  Life  Cycle  Software  Engineering  Center 


lea  .  Logistics  Evaluation  Agency 

LOB  .  Left-of-Baseline 

LOG . Line  of  Code 

LSVT  ....  Lead  Site  Verification  Test 
LUT . Limited  User  Test 

M/S  .  Modeling/Simulation 


MACOM  ...  Major  Command 

HAISRC  . .  Major  Automated  Information  System  Review 
Council 

MANPRINT  Manpower  and  Personnel  Integration 
MARC  ....  Manpower  Requirement  Criteria 
MATDEV  . .  Materiel  Developer 
MCIR  ....  Materiel  Change  Information  Report 


MNS  .  Mission  Needs  Statement 

MOA  .  Memorandum  of  Agreement 

MOE  .  Measure  of  Effectiveness 

MOP  .  Measure  of  Performance 

mot  .  Multi-service  Operational  Test 

MP . Management  Plan 
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MRRB  ....  Materiel  Release  Review  Board 

MS . Milestone 

MSC  .  Major  Subordinate  Command 

MSCR  ....  Materiel  System  Computer  Resources 

MSP  .  Mission  Support  Plan 

MTBF  .... -Mean  Time  Between  Mission  Failure 
MTTR  ....  Mean  Time  to  Restore 

NDI  .  Non-Development a 1  Item 

NHPP  ....  Nonhomogeneous  Poisson  Process 

NIST  ....  National  Institute  of  Science  and  Technology 

OA  .  Operational  Assessment 

OE  .  Operational  Evaluator 

OEC  .  Operational  Evaluation  Command 

OMA  .  Operations  and  Maintenance  Army 

0MB  .  Office  of  Management  and  Budget 

OMF  .  Operational  Mission  Failure 

OMS/MP  . .  Operational  Mode  Summary /Miss ion  Profile 

O&O  .  Operational  and  Organizational  Plan 

OPA  .  Operations /Procurement-Army 

OPTEC  . . .  Operational  Test  and  Evaluation  Command 

ORD  .  Operational  Requirements  Document 

OSUT  ....  On-site  User  Test 

OT . Operational  Test 

OT&E  ....  Operational  Test  and  Evaluation 

OTP . Outline  Test  Plan 

OTRR  ....  Operational  Test  Readiness  Review 
OTRS  ....  Operational  Test  Readiness  Statement 

PA  .  Proponent  Agency 

PCA  .....  Physical  Configuration  Audit 

PDL  .  Program  Design  Language 

PDR  .  Preliminary  Design  Review 

PDSS  ....  Post-Deployment  Software  Support 

PEO  .  Program  Executive  Officer 

PF  .  Program  Folder 

PM  .  Program/Project/Product  Manager 

POI  .  Programs  of  Instruction 

PPQT  ....  Pre-Production  Qualification  Test 

PR  .  Problem  Report 

PT . Test  Plan 

QA  .  Quality  Assurance 

QAD  .  Quality  Assurance  Directorate 

QIO  .  Quality  Improvement  Office 

QQPRI  ...  Qualitative  or  Quantitative  Personnel 
Requirements  Information 

RADC  ....  Rome  Air  Development  Center 
RAM  .  Random  Access  Memory 

RAM  .  Reliability,  Availability,  Maintainability 

RDTE  ....  Research,  Development,  Test  and  Evaluation 

RFP  .  Request  for  Proposal 

RGT  .  Reliability  Growth  Test 
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ROB  .  Right-of-Baseline 

ROC  .  Required  Operational  Capabilities 

RT  .  Test  Analysis  Report 

SAT  .  Software  Acceptance  Test 

SCART  . . . -Software  Corrective  Action  Review  Team 

SCP  .  Software  Change  Package 

SDC  .  Strategic  Defense  Command 

SDD  .  Software  Design  Document 

SDF  .  Software  Development  Folder 

SDIO  ....  Strategic  Defense  Initiative  Office 

SDL  .  Software  Development  Library 

SDP  .  Software  Development  Plan 

SDP . System  Design  Plan 

SDR  .  System  Design  Review 

SDT  .....  Software  Development  Test 

SEE  .  Software  Engineering  Environment 

SEI  .  Software  Engineering  Institute 

SIT . System  Integration  Test 

SLOC  ....  Source  Line  of  Code 

SOP  .  Standard  Operating  Procedures 

SOW . Statement  of  Work 

SPCR  ....  Software  Problem/ Change  Report 

SPM  .  Software  Programmer's  Manual 

SPR  .  System  Post-Deployment  Review 

SPS  .  Software  Product  Specification 

SQA . Software  Quality  Assurance 

SQPP  ....  Software  Quality  Program  Plan 

SRR  .  System  Requirements  Review 

SRS  .  Software  Requirements  Specification 

SRTM ^ . . .  Software  Requirements  Traceability  Matrix 

SS  .  System/ Subsystem  Specifications 

SSDD  ....  System/ Segment  Design  Document 
SSEB  ....  Source  Selection  Evaluation  Board 

SSG  .  Software  Sub-Group 

SSR  .  Software  Specification  Review 

SSS  .  System/ Segment  Specification 

SST  .  Supplemental  Site  Testing 

SSTP  ....  Software  Support  Transition  Plan 

STD  .  Software  Test  Description 

STEP  ....  Software  Test  and  Evaluation  Panel 
STP  .....  Software  Test  Plan 

STP  .  System  Integration  Plan 

STP . System  Test  Plan 

STR  .  Software  Test  Report 

STR  .  Software  Trouble  Report 

STSC  ....  Software  Technology  Support  Center 
SW  .  Software  (also  referenced  as  S/W) 

TCIC  ....  Technical  Critical  Issues  and  Criteria 
TD/CMS  ..  Technical  Data/Configuration  Management 
System 

TDP . Test  Design  Plan 

TE  .  Developmental  Evaluator 

T&E . Test  and  Evaluation 
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TECOM  ...  Test  and  Evaluation  Command 

TEMP  _  Test  and  Evaluation  Master  Plan 

TEMPEST  . . Control  of  Compromising  Emanations 

TEP  .  Test  and  Evaluation  Plan 

TER . Test  and  Evaluation  Report 

TEXCOM  . . .Test  and  Experimentation  Command 

TIR  . "Test  Incident  Reports 

TIWG  ....  Test  Integration  Working  Group 

TP . Test  Plan 

TQM  .  Total  Quality  Management 

TR . Test  Report 

TRADOC  . .  Training  and  Doctrine  Command 

TRR . Test  Readiness  Review 

TSARC  . . .  Test  Schedule  and  Review  Committee 

UFD  .....  Users'  Functional  Description 

UM . User  Manual 

US  ......  Software  Unit  Specification 

VCSA  ....  Vice  Chief  of  Staff  of  Army 

VDD  .  Version  Description  Document 

V&V  .  Verification  and  Validation 

WBS  .....  Work  Breakdown  Structure 
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